Gitlab and Github offers personal access tokens for git access over https. They are the only accepted method of authentication when you have Two-Factor Authentication (2FA) enabled. Since I have a Yubikey, I have to use a personal access token, if SSH is not viable, e.g. working in safe guarded environment. A token however has the advantage that it can expire, thus forcing me to exchange it more frequently to hinder attack scenarios.
Distributed Tracing with Jaeger by Uber Technologies is pretty impressive. As default you use Apache Cassandra as storage. Jaeger is also capable of using Elasticsearch 5/6. It took me some time and some code diving on github to find the respective options for Elasticsearch. But I finally got it together in this docker-compose.yml. My Elasticsearch Cluster runs with a commercial X-Pack license so we have to pass some authentication.
The QA (Quality Assurance) team use simulators like Astrex to check and test respective changes and features. I was asked if I could bring the simulator logs into our Elasticsearch, for a real time purpose. Tailing log files is still difficult, except if you can use bash.
A customer of mine, requires xml data as separate field data for further investigation. The data itself is part of a log message that is processed by Logstash. Logstash provides the powerful XML filter plugin for further parsing.
By default, Docker captures the standard output (and standard error) of all your containers, and writes them in files using the JSON format. It is advised to set a max size, otherwise you will run out of disk space. Having unified logging with Elasticsearch allows you to investigate logs in a single point of view. Sending the logs to Elasticsearch from the Docker containers is quite easy. Fluentd is a data collector, which a Docker container can use by omitting the option --log-driver=fluentd.
Storing data in Elasticsearch with city names, offers the capability to display in Kibana the distribution of the data in geographical map. To use that feature, you have to declare a geo_point type in your index mapping. I named the field location. To translate the city names to their respective geo point I use the logstash translate filter. Having a small dictionary, logstash will just take the value for your input city. You could also use zip codes, but this would require a more detailed data source. For the demonstration of the translation plugin it is sufficient.
At work I still run the Elasticsearch Cluster in version 5.6.4. While I’m eager to upgrade and keep up the pace, I don’t always have the chance to upgrade immediately. A customer of mine needed a small set of data in Excel. Elasticsearch 6 or moreover Kibana 6 offers the CSV export in the X-Pack extensions. To use that functionality, I needed to export a fragment of desired data from my production cluster. Since the Reindex API allows us to read data from remote and write it, I simply ramped up my private cluster in v 6.1.1 with Docker and started the reindexing.
Providing a HTTP health check service with Nginx, is straightforward. If you do ensure that Nginx closes the HTTP connection instead keeping it alive. The basic option therefore is:
Performing a reindex job in Elasticsearch gives you the time the job took.