Working in the financial business requires to have the currency code master-data accessible for various reasons. The ISO 4217 currency codes can be obtained from the ISO Organization website. This post uses Logstash and the csv plugin to process the data and import it into Elasticsearch. Elasticsearch itself provides the REST interface, so every micro-service or web service can access the desired data.
Elasticsearch Alerting with X-Pack (formerly known as Watcher), writes it watch executions in a daily indices. If you don’t keep an eye on that, you use a lot of shards on small indices. Curator offers the capability of the reindex action, i.e. write data from a daily index into a month or year index. This post contains an example for Elasticsearch v5.4.3 and Elasticsearch Curator v5.1.1.
Monit allows start, stop and restart program instructions in process checks. If you restart with monit, no alarm or warning is triggered then. For instance you need to restart an application like logstash for configuration changes.
Working with Elasticsearch Watcher enables you to put a cron schedule into the trigger. This is no ordinary linux cron expression. It looks like Quartz. If you want to test the correctness of the cron, you use the shipped utility croneval in the installed X-Pack directory.
HTTP basic authentication does base64 encoding. base64 exists on nearly every Linux distribution.
TL,DR (Too long, Don’t read.). If you have a dedicated monitoring cluster for your Elasticsearch clusters, you should at least monitor the memory usage of each node. This is very helpful. Instead of fetching the data from the cluster within, we query the monitoring cluster remotely. This watch was created on Elasticsearch with X-Pack v5.4.3. Pay attention, that some values are tweaked and not realistic for production scenarios. The interval for instance is set to 6 hours, since we will execute this watch manually. We also choose 60% as threshold. 75% or 80% would be more realistic for warning scenarios.
gosu is an essential help for dockerized applications. Following recipe is from daily work. I always have some connectivity issues due to security precautions. This recipe works especially behind corporate firewalls with a http proxy.
Working with Kerberos these days. Some recipes for dealing with Kerberos. There are several open source implementations out there.
Running a Elasticsearch cluster can be an easy task. If you have the need to store data for a long time, but you know the data is infrequently requested, you may think of a hot-warm architecture. This post is a brief summary of my setup for my company at work. At that time Elasticsearch v5.4.1 was running.