Loading...

Using Sidecar Container for Elasticsearch Configuration

Applications shipped in Docker containers are a major game changer, especially having a Elasticsearch cluster. My production cluster consists of 11 nodes. In the core, Elasticsearch is the same. Each node though has its specific configuration, settings and purpose. On top of that, Elasticsearch X-Pack Security in Version 6 requires that the communication within the cluster must run encrypted. This is accomplished by SSL certificates. Each node has its own private key and certificate. So I was facing with the problem, how to ship the node specific parts along with the core elasticsearch container. Use the core container as baseline and copy the configuration and certificate into the container? This would resolve in 11 specific images. Not in the spirit of reusability though. :thinking: The better approach or answer came by remembering the tech talk Docker Patterns by Roland Huss, given at the Java Conference (Javaland 2016). Use a configuration container as a sidecar!

Concept

The basic question to the pattern is: How to configure containerized applications for different environments ?

Solution Patterns:

  • Env-Var Configuration
  • Configuration Container
  • Configuration Service

ENV-VAR Configuration

The bullet points from Docker Patterns:

  • Standard configuration method for Docker container
  • Specified during build or run time
  • Universal

We can define environment variables and configure the docker application to use them. Environment variables can be overwritten by passing them.

Elasticsearch already does that, for example:

sudo docker run -it \
 -v /etc/timezone:/etc/timezone -v /etc/localtime:/etc/localtime \
 -v /srv/nas/elasticsearch/config/test:/opt/elasticsearch/config \
 -v /var/opt/elasticsearch:/var/opt/elasticsearch  \
 -v /var/log/elasticsearch:/var/log/elasticsearch  \
 -v /srv/nas/elasticsearch/backup:/srv/nas/elasticsearch/backup \ 
 --name="elasticsearch" \
 --cap-add=IPC_LOCK --ulimit memlock=-1:-1 --ulimit nofile=65536:65536 \
 -e TZ=Europe/Zurich \
 -e ES_JAVA_OPTS="-Xms8g -Xmx8g" \
 elasticsearch:latest bin/elasticsearch \
 -E network.host=node-1 \
 -E network.publish_host=node-1 \
 -E node.name=alpha \
 -E path.conf=/opt/elasticsearch/config

We have multiple environment bypasses in above example.

  • For instance we set docker environment variables like timezone (TZ).
  • The -E node.name=alpha is an Elasticsearch argument.

The elasticsearch node configuration was identical. Only the node specific information were provided.

This was done by me in the past. It worked until the requirement for node certificates. This doesn’t work for certificate files. Let’s take a look on the next approach.

Configuration container

The bullet points from Docker Patterns:

  • Configuration in extra Docker container
  • Volume linked during runtime

Pros

  • Flexible
  • Explicit

Cons

  • Static
  • Maintenance overhead
  • Custom file layout

The basic idea is to run 11 elasticsearch containers and just attaching or linking them to the configuration container (symbolically as sidecar).

Summary

The third approach is using some kind of central configuration registry or service. Consul, etcd or Apache Zookeeper are viable solutions, but in my scenario with Elasticsearch not applicable.

So Sidecar it is!

Sidecar

This pattern is named Sidecar because it resembles a sidecar attached to a motorcycle. In the pattern, the sidecar is attached to a parent application and provides supporting features for the application. The sidecar also shares the same lifecycle as the parent application, being created and retired alongside the parent. The sidecar pattern is sometimes referred to as the sidekick pattern and is a decomposition pattern.

Motorcycle with Sidecar

Creating the Sidecar Container

I will demonstrate how I applied the sidecar pattern for my elasticsearch test-cluster of 3 nodes.

First my project layout:

$ ll
total 78
-rw-r--r-- 1 tan 1049089   463 Mar  1 12:50 Dockerfile
drwxr-xr-x 1 tan 1049089     0 Feb 28 18:01 node-1/
drwxr-xr-x 1 tan 1049089     0 Feb 28 18:00 node-2/
drwxr-xr-x 1 tan 1049089     0 Feb 28 18:00 node-3/
-rw-r--r-- 1 tan 1049089  1513 Feb 28 17:28 Jenkinsfile

Each node has following node specific configuration. Mandatory for Elasticsearch is elasticsearch.yml and certs folder for X-Pack security.

$ ls -lR node-1/
itu/:
total 20
drwxr-xr-x 1 tan 1049089    0 Mar  1 12:57 certs/
-rw-r--r-- 1 tan 1049089 1700 Feb 28 17:12 elasticsearch.yml
-rw-r--r-- 1 tan 1049089 1920 Feb 28 17:02 jvm.options
-rw-r--r-- 1 tan 1049089 4459 Feb 28 16:53 log4j2.properties
drwxr-xr-x 1 tan 1049089    0 Feb 28 18:02 scripts/

node-1/certs:
total 16
-rw-r--r-- 1 tan 1049089 1314 Feb 28 16:53 ca.crt
-rw-r--r-- 1 tan 1049089 2985 Feb 28 16:53 ElasticCloudCaChain.pem
-rw-r--r-- 1 tan 1049089 1346 Feb 28 17:10 node-1.crt
-rw-r--r-- 1 tan 1049089 1679 Feb 28 17:10 node-1.key

node-1/scripts:
total 1
-rw-r--r-- 1 tan 1049089 60 Feb 28 18:02 README.md

Dockerfile

The Dockerfile to build the sidecar container.

FROM alpine

MAINTAINER Vinh Nguyen <le-mapper@cinhtau.net>
LABEL es.version="5.6.8"

ENV http_proxy=${http_proxy:-http://localhost:3128} \
    https_proxy=${https_proxy:-https://localhost:3128}

RUN apk --no-cache add shadow && \
    adduser -D -u 1000 elasticsearch

COPY . /config/

RUN chown -R elasticsearch:elasticsearch /config

VOLUME /config

The COPY command copies the configuration to the container /config/ directory. Pay attention to ignore unwanted files in the .dockerignore file.

Check contents

You can inspect the sidecar container by executing the list directory command ls. After the execution, the docker container is automatically removed.

docker run -it --volumes-from es_config --rm alpine /bin/sh -c "ls -lR /config"

Deployment

On the docker host, we don’t need to run the sidecar container. If we just create the container the docker volume is available.

docker pull elasticsearch-config-test && \
docker create --name es_config elasticsearch-config-test /bin/true

Usage

Using Elasticsearch with configuration: To use the configuration volume from our sidecar container, omit this option --volumes-from es_config, where es_config is the name of the docker container.

sudo docker run -d \
 --volumes-from es_config \
 -v /etc/timezone:/etc/timezone -v /etc/localtime:/etc/localtime \ 
 -v /var/opt/elasticsearch:/var/opt/elasticsearch  \
 -v /var/log/elasticsearch:/var/log/elasticsearch  \
 -v /srv/nas/elasticsearch/backup:/srv/nas/elasticsearch/backup \ 
 -v /srv/nas/elasticsearch/security:/config/node-1/x-pack \
 -v /srv/nas/elasticsearch/ingest-geoip:/config/node-1/ingest-geoip \
 --log-driver none \
 --net=host --name="elasticsearch" \
 --cap-add=IPC_LOCK --ulimit memlock=-1:-1 --ulimit nofile=65536:65536 \
 --restart on-failure:10 \
 -e TZ=Europe/Zurich \
 -e ES_JAVA_OPTS="-Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128 -Dhttps.proxyHost=localhost -Dhttps.proxyPort=3128 -Dhttp.nonProxyHosts=\"localhost|127.0.0.1|.cinhtau.net|.aws.com\"" \
 elasticsearch:latest bin/elasticsearch \
 -E path.conf=/config/node-1

Container Lifecycle

The sidecar container wil be removed, if you have some cleanup, but the volume still exist, since it used in by the elasticsearch application container. A docker inspect of the application container will give you the name of the used volume.

docker inspect elasticsearch

Output shortened:

{
  "Mounts": [
    {
      "Source": "/srv/nas/six/fo/elasticsearch/security",
      "Destination": "/config/itu/x-pack",
      "Mode": "",
      "RW": true,
      "Propagation": "rslave"
    },
    {
      "Source": "/etc/timezone",
      "Destination": "/etc/timezone",
      "Mode": "",
      "RW": true,
      "Propagation": "rslave"
    },  
    {
      "Name": "f89912de22e2b34170e0b331c8a5e25b00f921f4e2417c6b140382389fadee7e",
      "Source": "/var/lib/docker/volumes/f89912de22e2b34170e0b331c8a5e25b00f921f4e2417c6b140382389fadee7e/_data",
      "Destination": "/config",
      "Driver": "local",
      "Mode": "",
      "RW": true,
      "Propagation": ""
    }]
}

The volume name f89912de22e2b34170e0b331c8a5e25b00f921f4e2417c6b140382389fadee7e is used.

To check:

docker volume inspect f89912de22e2b34170e0b331c8a5e25b00f921f4e2417c6b140382389fadee7e
[
    {
        "Name": "f89912de22e2b34170e0b331c8a5e25b00f921f4e2417c6b140382389fadee7e",
        "Driver": "local",
        "Mountpoint": "/var/lib/docker/volumes/f89912de22e2b34170e0b331c8a5e25b00f921f4e2417c6b140382389fadee7e/_data"
    }
]

As you can see the volume name is hard for humans to remember. Docker offers to create named volumes as cherry on top.

# create named volume
docker volume create --name es_config_test

# populate volume use one-shot sidecar
docker run --rm -v es_config_test:/config elasticsearch-config-test /bin/true

# check volume using shell
docker run --rm -v es_config_test:/config alpine /bin/sh -c 'ls -laR /config'

# show volume
docker volume inspect es_config_test

Summary

  • The sidecar pattern is very useful.
  • Sidecar containers have multiple purposes:
    • Configuration
    • Proxy (using Nginx as Load Balancer or Proxy)
    • Logging (running log shipper for centralized logging)
    • every other abstraction
  • A sidecar container lifecycle is tight to its application container(s).

Comments


Leave a comment