[Docker-elk] <최신 버전> ec2에서 docker-elk 환경 구성이 안된다면?!

본 포스팅은 ec2에서 docker-elk 환경 구성 시 발생하는 에러를 해결하는 방법을 다룹니다.

문제 상황 발생 배경 설명

• local machine(m1 mac)과 동일하게 https://github.com/deviantony/docker-elk 에서 설치를 진행함
• 운영서버에서 nginx를 통해서 리버스 프록시를 설정한 후, 아무런 설정을 건드리지 않았는데, 5601 포트로 접속하였을 때, 401에러를 반환
• 401에러를 반환하는 것을 해결하는 것이 핵심인 줄 알았는데, 모니터링 서버의 kibana의 기본포트인 5601 포트로도 접속이 안되는 것을 발견

발생한 에러들

# kibana
[2024-04-07T03:38:31.593+00:00][INFO ][plugins.assetManager] Server is NOT enabled
[2024-04-07T03:38:33.000+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. connect ECONNREFUSED 172.21.0.2:9200
[2024-04-07T03:38:38.741+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/node_modules/@kbn/screenshotting-plugin/chromium/headless_shell-linux_x64/headless_shell
[2024-04-07T03:39:05.672+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception
        Root causes:
                security_exception: unable to authenticate user [kibana_system] for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]

# log stash
[2024-04-07T03:39:02,850][INFO ][logstash.outputs.elasticsearch][main] Failed to perform request {:message=>"Connect to elasticsearch:9200 [elasticsearch/172.21.0.2] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to elasticsearch:9200 [elasticsearch/172.21.0.2] failed: Connection refused>}
[2024-04-07T03:39:02,854][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://logstash_internal:xxxxxx@elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/172.21.0.2] failed: Connection refused"}
[2024-04-07T03:39:08,278][WARN ][logstash.outputs.elasticsearch][main] Health check failed {:code=>401, :url=>http://elasticsearch:9200/, :message=>"Got response code '401' contacting Elasticsearch at URL 'http://elasticsearch:9200/'"}
[2024-04-07T03:39:08,280][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://logstash_internal:xxxxxx@elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '401' contacting Elasticsearch at URL 'http://elasticsearch:9200/'"}
[2024-04-07T03:39:19,707][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://logstash_internal:xxxxxx@elasticsearch:9200/"}
[2024-04-07T03:39:19,836][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.13.0) {:es_version=>8}
[2024-04-07T03:39:19,838][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}

# elastic search
{"@timestamp":"2024-04-07T03:39:09.686Z", "log.level": "INFO", "message":"Authentication of [kibana_system] was terminated by realm [reserved] - failed to authenticate user [kibana_system]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch][transport_worker][T#1]","log.logger":"org.elasticsearch.xpack.security.authc.RealmsAuthenticator","trace.id":"bf17776cbfc61b625d630d43f3a18d6b","elasticsearch.cluster.uuid":"ZMsf_TrCSsW16juRyf2c-w","elasticsearch.node.id":"_OkwFfHOTlqvx1VSTmCImg","elasticsearch.node.name":"elasticsearch","elasticsearch.cluster.name":"docker-cluster"}

해결 방법

• 대충 에러들을 종합해보면 elastic search에 접근하지 못하고 있다는 것인데 그래서 여러 포스팅들을 보면서 elasticsearch, kibana, logstash, docker-compose, .env의 설정을 만져보는등의 시도를 함.
  • 하지만 관련 포스팅이 너무 옛날 것이라 설정하는 법이 바뀌었나 ? 그런 생각을 하였음.
• 하지만 문제는 그것이 아니었다.... elastic search의 기본 사양이 메모리 4GB였던 것... 처음 small로 instance를 생성하고 swap 메모리 2G를 해주어서 괜찮을 거라고 생각했지만 기본 메모리가 4GB이상이여야 했었던 것....
• 따라서 ec2 인스턴스를 기본 메모리가 4GB인 medium으로 늘려주니 마법같이 포트로 접속 성공!