This post is older than a year. Consider some information might not be accurate anymore.
Used: elasticsearch 6.1.2 fluentd 0.12 docker 17.12
By default, Docker captures the standard output (and standard error) of all your containers, and writes them in files using the JSON format. It is advised to set a max size, otherwise you will run out of disk space. Having unified logging with Elasticsearch allows you to investigate logs in a single point of view. Sending the logs to Elasticsearch from the Docker containers is quite easy. Fluentd is a data collector, which a Docker container can use by omitting the option
docker run --log-driver=fluentd ubuntu echo 'Hello Fluentd!'
All we have to do, is to run Fluentd with the Elasticsearch output plugin. We can run Fluentd with Docker itself . Build fluentd Docker image with Elasticsearch plugin. This is a slight modified version of the official Docker image.
FROM fluent/fluentd:v0.12-onbuild ENV http_proxy http://10.0.2.2:3128 ENV https_proxy https://10.0.2.2:3128 ENV no_proxy localhost,my-elasticsearch-cluster USER root RUN adduser -S fluent && apk add --update --virtual .build-deps \ sudo build-base ruby-dev \ && gem install \ fluent-plugin-elasticsearch \ && gem sources --clear-all \ && apk del .build-deps \ && rm -rf /var/cache/apk/* \ /home/fluent/.gem/ruby/2.3.0/cache/*.gem
The fluentd configuration:
<source> @type forward port 24224 bind 0.0.0.0 </source> <match docker.consul> @type elasticsearch host my-es-cluster user cinhtau password "lemapper" port 9200 tag_key @le_mapper flush_interval 5s # for testing logstash_format true logstash_prefix consul type_name doc pipeline consul </match> <match docker.*> @type elasticsearch host my-es-cluster user cinhtau password "lemapper" port 9200 tag_key @le_mapper flush_interval 5s logstash_format true logstash_prefix docker type_name doc </match>
Consul logs are ingested into the
consul index name pattern which matches the
docker.consul tag. Other output goes to the
The tag for fluentd is omitted by the run command:
docker run --log-driver=fluentd --log-opt fluentd-tag=docker.consul consul