Docker Centralized Logging with ELK Stack

As your infrastructure grows, it becomes crucial to have robots and a reliable centralized logging system. Log centralization is becoming a key aspect of a variety of IT tasks and provides you with an overview of your entire system.

The best solution is to aggregate the logs from all containers, which is enriched with metadata so that it provides you with better traceability options and comes with awesome community support. This is where ELK Stack comes into the picture. ELK, also known as Elastic stack, is a combination of modern open-source tools like ElasticSearch, Logstash, and Kibana. It is a complete end-to-end log analysis solution you can use for your system.

Each component has its defined role to play: ElasticSearch is best in storing the raw logs, Logstash helps to collect and transform the logs into a consistent format, and Kibana adds a great visualization layer and helps you to manage your system in a user-friendly manner.

In this guide, you will learn how to deploy ELK and start aggregating container logs. Here we are going to combine ELK with Filebeat to aggregate the container logs. For this, we are going to build a custom Docker image.

Step 1: Configuring Filebeat

Let’s begin with the Filebeat configuration. First, you have to create a Dockerfile to create an image:

$ mkdir filebeat_docker && cd $_$ touch Dockerfile && nano Dockerfile

Now, open the Dockerfile in your preferred text editor, and copy/paste below mentioned lines:

FROM docker.elastic.co/beats/filebeat:7.5.1 COPY filebeat.yml /usr/share/filebeat/filebeat.ymlUSER rootRUN mkdir /usr/share/filebeat/dockerlogsRUN chown -R root /usr/share/filebeat/RUN chmod -R go-w /usr/share/filebeat/

In filebeat_docker directory, create a filebeat.yml file that contains configuration for Filebeat. For this guide, we are going to use a minimal filebeat.yml file.

filebeat.inputs:  - type: docker    containers:      path: "/usr/share/dockerlogs/data"      stream: "stdout"      ids:        - "*"      cri.parse_flags: true      combine_partial: true      exclude_files: ['\.gz$'] processors:  - add_docker_metadata:      host: "unix:///var/run/docker.sock" filebeat.config.modules:  path: ${path.config}/modules.d/*.yml  reload.enabled: false output.logstash:  hosts: ["127.0.0.1:5044"] log files:logging.level: errorlogging.to_files: falselogging.to_syslog: falseloggins.metrice.enabled: falselogging.files:  path: /var/log/filebeat  name: filebeat  keepfiles: 7  permissions: 0644ssl.verification_mode: none

Now, it’s time to create the Filebeat Docker image:

$ docker build -t filebeatimage .Sending build context to Docker daemon  3.584kBStep 1/6 : FROM docker.elastic.co/beats/filebeat:7.5.17.5.1: Pulling from beats/filebeatc808caf183b6: Already exists a07383b84bc8: Pull complete a3c8dd4531b4: Pull complete 5547f4a87d0c: Pull complete d68e041d92cd: Pull complete 7cfb3f76a272: Pull complete 748d7fe7bf07: Pull complete Digest: sha256:68d87ae7e7bb99832187f8ed5931cd253d7a6fd816a4bf6a077519c8553074e4Status: Downloaded newer image for docker.elastic.co/beats/filebeat:7.5.1 ---> 00c5b17745d1Step 2/6 : COPY filebeat.yml /usr/share/filebeat/filebeat.yml ---> f6b75829d8d6Step 3/6 : USER root ---> Running in 262c41d7ce58Removing intermediate container 262c41d7ce58 ---> 1ffcda8f39cfStep 4/6 : RUN mkdir /usr/share/filebeat/dockerlogs ---> Running in 8612b1895ac7Removing intermediate container 8612b1895ac7 ---> 483d29e65dc7Step 5/6 : RUN chown -R root /usr/share/filebeat/ ---> Running in 4a6ad8b22705Removing intermediate container 4a6ad8b22705 ---> b779a9da7ac9Step 6/6 : RUN chmod -R go-w /usr/share/filebeat/ ---> Running in bb9638d12090Removing intermediate container bb9638d12090 ---> 85ec125594eeSuccessfully built 85ec125594eeSuccessfully tagged filebeatimage:latest

To verify if the image was built successfully:

$ docker imagesREPOSITORY      TAG           IMAGE ID            CREATED             SIZEfilebeatimage   latest        85ec125594ee        7 seconds ago       514MB

For filebeat_elk container, you have created two mounts using the parameter -v;

• /var/lib/docker/containers:/usr/share/dockerlogs/data: You have mapped host machine docker logs which resides in /var/lib/docker/containers to /usr/share/dockerlogs/data inside the docker container. Note that you have used :ro which denotes that has read-only permission.
• Whereas, /var/run/docker.sock is bind with Filebeat container’s Docker daemon, which allows Filebeat container to gather the Docker’s metadata and container logs entries.

Filebeat installation via DEB:

There is an alternate way to install Filebeat in your host machine. At the time of writing, Filebeat version is 7.5.1 you can download the latest version of filebeat from here.

To install the downloaded .deb file:

$ wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.5.1-amd64.deb$ sudo dpkg -i filebeat-7.5.1-amd64.deb

You can find the configuration file in /etc/filebeat/filebeat.yml directory.

Step 2: Configuring ELK or Elastic Stack

You can either use a remote server to host your ELK stack or can launch containers within your existing system.

Before you get going, make sure that the following ports are listening:

• Elasticsearch – Port 9200 and Port 9300
• Logstash – Port 5044
• Kibana – Port 5601

ElasticSearch:

We are going to use the latest official image of Elasticsearch as of now. So begin by pulling the image from Docker Hub:

$ docker pull docker.elastic.co/elasticsearch/elasticsearch:7.5.17.5.1: Pulling from elasticsearch/elasticsearchc808caf183b6: Already exists 05ff3f896999: Pull complete 82fb7fb0a94e: Pull complete c4d0024708f4: Pull complete 136650a16cfe: Pull complete 968db096c092: Pull complete 42547e91692f: Pull complete Digest: sha256:b0960105e830085acbb1f9c8001f58626506ce118f33816ea5d38c772bfc7e6cStatus: Downloaded newer image for docker.elastic.co/elasticsearch/elasticsearch:7.5.1docker.elastic.co/elasticsearch/elasticsearch:7.5.1

Now, create a directory name as docker_elk, where all your configuration files and Dockerfile will reside:

$ mkdir docker_elk && cd $_

Inside docker_elk, create another directory for elasticsearch and create a Dockerfile and elasticsearch.yml files:

$ mkdir elasticsearch && cd $_$ touch Dockerfile && touch elasticsearch.yml

Open elasticsearch.yml file in your preferred text editor and copy the configuration setting as it is:

---cluster.name: "docker-cluster"network.host: 0.0.0.0 xpack.license.self_generated.type: basicxpack.security.enabled: truexpack.monitoring.collection.enabled: true

Note that you can set xpack.license.self_generated.type from basic to trial if you wish to evaluate the commercial feature of x-pack for 30 days.

Open Dockerfile in your preferred text editor and copy the below-mentioned lines and paste it as it is:

FROM docker.elastic.co/elasticsearch/elasticsearch:7.5.1COPY --chown=elasticsearch:elasticsearch ./elasticsearch.yml /usr/share/elasticsearch/config/

The command chown is to change the file owner to elasticsearch as of other files in container.

Kibana:

Now, you are going to setup Dockerfile for Kibana, and again you have to pull the latest image from the Elastic Docker registry:

$ docker pull docker.elastic.co/kibana/kibana:7.5.17.5.1: Pulling from kibana/kibanac808caf183b6: Already exists e12a414b7b04: Pull complete 20714d0b39d8: Pull complete 393e0a5bccf2: Pull complete b142626e938b: Pull complete b28e35a143ca: Pull complete 728725922476: Pull complete 96692e1a8406: Pull complete e4c3cbe1dbbe: Pull complete bb6fc46a19d1: Pull complete Digest: sha256:12b5e37e0f960108750e84f6b2f8acce409e01399992636b2a47d88bbc7c2611Status: Downloaded newer image for docker.elastic.co/kibana/kibana:7.5.1docker.elastic.co/kibana/kibana:7.5.1

Inside your docker_elk, create a directory, and inside of it, you have to create a Dockerfile and kibana.yml files:

$ mkdir kibana && cd $_$ touch Dockerfile && touch kibana.yml

kibana.yml will consist of follow configurations. Note that you have to change the values of elasticsearch.user and elasticsearch.password:

---server.name: kibanaserver.host: "0"elasticsearch.hosts: [ "http://elasticsearch:9200" ]xpack.monitoring.ui.container.elasticsearch.enabled: true elasticsearch.username: elasticelasticsearch.password: yourstrongpasswordhere

Whereas, in Dockerfile, will look something like this:

FROM docker.elastic.co/kibana/kibana:7.5.1COPY ./kibana.yml /usr/share/kibana/config/

Logstash:

Container image for Logstash is available from the Elastic Docker registry. Again at the time of writing current version is 7.5.1, you can find latest version of Logstash here.

$ docker pull docker.elastic.co/logstash/logstash:7.5.17.5.1: Pulling from logstash/logstashc808caf183b6: Already exists 7c07521065ed: Pull complete d0d212a3b734: Pull complete 418bd04a229b: Pull complete b22f374f97b1: Pull complete b65908943591: Pull complete 2ee12bfc6e9c: Pull complete 309701bd1d88: Pull complete b3555469618d: Pull complete 2834c4c48906: Pull complete bae432e5da20: Pull complete Digest: sha256:5bc89224f65459072931bc782943a931f13b92a1a060261741897e724996ac1aStatus: Downloaded newer image for docker.elastic.co/logstash/logstash:7.5.1docker.elastic.co/logstash/logstash:7.5.1

Now, create a directory for Logstash inside docker_elk and add necessary files as shown below:

$ mkdir logstash && cd $_$ touch Dockerfile && touch logstash.yml

Copy below mentioned line into logstash.yml. Make sure that you enter the right username and password in xpack.monitoring.elasticsearch.username and xpack.monitoring.elasticsearch.password respectively:

---http.host: "0.0.0.0"xpack.monitoring.elasticsearch.hosts: [ "http://elasticsearch:9200" ] xpack.monitoring.enabled: truexpack.monitoring.elasticsearch.username: elasticxpack.monitoring.elasticsearch.password: yourstrongpasswordhere

Now, add following lines into your Dockerfile:

FROM docker.elastic.co/logstash/logstash:7.5.1COPY ./logstash.yml /usr/share/logstash/config/COPY ./logstash.conf /usr/share/logstash/pipeline/

Apart from this, you have to create a logstash.conf file. Here in elasticsearch reference you will find host, user and password, make sure you change the values as per your system:

input {    tcp {    port => 5000    codec => json  }}output {  elasticsearch {    hosts => "elasticsearch:9200"    user => elastic    password => yourstrongpasswordhere  }}

As you are through with the setup of your stack’s components, the directory structure of your project should should look something like this:

.├── elasticsearch│   ├── Dockerfile│   └── elasticsearch.yml├── kibana│   ├── Dockerfile│   └── kibana.yml└── logstash    ├── Dockerfile    ├── logstash.conf    └── logstash.yml 3 directories, 7 files

Now, it’s time to create a Docker Compose file, which will let you run the stack.

Step 3: Docker Compose

Create a docker-compose.yml file in the docker_elk directory. Here you are going to define and run your multi-container application consist of Elasticsearch, Kibana, and Logstash.

You can copy the below-mentioned context in your docker-compose.yml file. Please make sure that you change the ELASTIC_PASSWORD and ES_JAVA_OPTS values. For this guide, ES_JAVA_OPTS is set to 256 MB, but in real world scenarios you might want to increase the heap size as per requirement.

version: '3.2' services: elasticsearch:   build:     context: elasticsearch/   volumes:     - type: volume       source: elasticsearch       target: /usr/share/elasticsearch/data   ports:     - "9200:9200"     - "9300:9300"   environment:     ES_JAVA_OPTS: "-Xmx256m -Xms256m"     ELASTIC_PASSWORD: yourstrongpasswordhere     discovery.type: single-node   networks:     - elk_stack  logstash:   build:     context: logstash/   ports:     - "5000:5000"     - "9600:9600"   environment:     LS_JAVA_OPTS: "-Xmx256m -Xms256m"   networks:     - elk_stack   depends_on:     - elasticsearch  kibana:   build:     context: kibana/   ports:     - "5601:5601"   networks:     - elk_stack   depends_on:     - elasticsearch networks: elk_stack:   driver: bridge volumes: elasticsearch:

Now, to build the ELK stack, you have to run the following command in your docker_elk directory:

$ docker-compose up -dStarting elastic_elk ... doneStarting kibana_elk   ... doneStarting logstash_elk ... done

To ensure that the pipeline is working all fine, run the following command to see the Elasticsearch indices:

$ curl 'localhost:9200/_cat/indices?v' -u elastic:yourstrongpasswordherehealth status index                             uuid                   pri rep docs.count docs.deleted store.size pri.store.sizegreen  open   .triggered_watches                m-l01yMmT7y2PYU4mZ6-RA   1   0          0            0      6.5kb          6.5kbgreen  open   .watcher-history-10-2020.01.10    SX3iYGedRKKCC6JLx_W8fA   1   0       1523            0        2mb            2mbgreen  open   .management-beats                 ThHV2q9iSfiYo__s2rouIw   1   0          6            1     40.5kb         40.5kbgreen  open   .ml-annotations-6                 PwK7Zuw7RjytoWFuCCulJg   1   0          0            0       283b           283bgreen  open   .monitoring-kibana-7-2020.01.10   8xVnx0ksTHShds7yDlHQvw   1   0       1006            0    385.4kb        385.4kbgreen  open   .monitoring-es-7-2020.01.10       CZd89LiNS7q-RepP5ZWhEQ   1   0      36412          340     16.4mb         16.4mbgreen  open   .apm-agent-configuration          e7PRBda_QdGrWtV6KECsMA   1   0          0            0       283b           283bgreen  open   .ml-anomalies-shared              MddTZQ7-QBaHNTSmOtUqiQ   1   0          1            0      5.5kb          5.5kbgreen  open   .kibana_1                         akgBeG32QcS7AhjBOed3LA   1   0       1105           28    687.1kb        687.1kbgreen  open   .ml-config                        CTLI-eNdTkyBmgLj3JVrEA   1   0         22            0     56.6kb         56.6kbgreen  open   .ml-state                         gKx28CMGQiuZyx82bNUoYg   1   0          0            0       283b           283bgreen  open   .security-7                       krH4NlJeThyQRA-hwhPXEA   1   0         36            0     83.6kb         83.6kbgreen  open   .logstash                         7wxswFtbR3eepuWZHEIR9w   1   0          0            0       281b           281bgreen  open   .kibana_task_manager_1            ft60q2R8R8-nviAyc0caoQ   1   0          2            1     16.2kb         16.2kbyellow open   filebeat-7.5.1-2020.01.10-000001  1-RGhyG9Tf-wGcepQ49mmg   1   1          0            0       283b           283bgreen  open   .monitoring-alerts-7              TLxewhFyTKycI9IsjX0iVg   1   0          6            0     40.9kb         40.9kbgreen  open   .monitoring-logstash-7-2020.01.10 dc_S5BhsRNuukwTxbrxvLw   1   0       4774            0      1.1mb          1.1mbgreen  open   .watches                          x7QAcAQZTrab-pQuvonXpg   1   0          6            6    120.2kb        120.2kbgreen  open   .ml-notifications-000001          vFYzmHorTVKZplMuW7VSmw   1   0         52            0     81.6kb         81.6kb

Now, it is time to pay a visit to our Kibana dashboard. Open your browser and enter the URL http://your-ip-addr-here:5601. Now enter the predefined username and password; in our case, it is elastic and yourstrongpasswordhere, respectively.

In your Kibana dashboard, go to the Management tab, and under Kibana, click on Index Patterns. In the first row, you will find the filebeat-* index, which already has been identified by Kibana.

Now, go to the Discover tag on the Kibana dashboard and view your container logs along with the metadata under the selected index pattern, which could look something like this:

Conclusion:

You have now installed and configured the ELK Stack on your host machine, which is going to collect the raw log from your Docker into the stack that later can be analyzed or can be used to debug applications.

Explore Gcore Container as a Service