A Journey of a Thousand Miles Begins with a Single Step

Reindex data in Elasticsearch

Today we have reached more than 3000 shards in our elasticsearch clusters. Digging a little deeper, that is definitely too much. Since a shard (primary or replica) is a Lucene index, it consumes file handles, memory, and CPU resources. Each search request will touch a copy of every shard in the index, which isn’t a problem when the shards are spread across several nodes. Contention arises and performance decreases when the shards are competing for the same hardware resources. :- If you keep the logstash daily default, you will come in the situation very soon. I choose now a monthly basis. The latest 2-3 months are kept and old indices are deleted. The outcome is to merge all daily indices of a month to a big index. Therefore the Elasticsearch Reindex API is very useful.

Read more

Handling logstash input multiline codec

The multiline codec will collapse multiline messages and merge them into a single event. The default limit is 500 lines. If there are over 500 lines appended, the multiline codec split the message to the next 500 lines and so forth. This post demonstrates how to deal with this situation. Elasticsearch receives in tags the multiline_codec_max_lines_reached.

Read more

Localisation problem while deinstalling Oracle 11g on Windows

Well I stumbled over a funny fact, that Oracle could not handle localization properly. They ask in german for a choice with ja/nein means yes/no. See for yourself to tell the deinstaller my choice.

Read more

NonStop SQL MX with IntelliJ or DataGrip

NonStop SQL is a commercial relational database management system that is designed for fault tolerance and scalability for the HP NonStop. The latest version of the product is SQL MX 3.2.1 which was released in February 2013. This post describes how to setup IntelliJ or respective DataGrip to work with SQL MX.

Read more

Migrate elasticsearch indices from different clusters with logstash

I got an exceptional case in the office. Some application logs, which belongs to a dev and testing environment, were stored or reported in the elasticsearch production cluster. Therefore a cleanup or migration was necessary.

Read more

Housekeeping of log files

Writing software also results in writing application logs. Therefore log rotating or house keeping is essential to free the space of old and unused log files. While Linux provides logrotate, you may run into the situations that you aren’t root or an user with root permissions and are not eligible to use logrotate. A simple shell script will also provide the essential cleanup.

Read more

Using dictionaries in bash 4

Bash 4 supports dictionaries, hash tables or associative arrays. I was in need of that feature writing an logstash script, working with environment variables in logstash itself. A simple demonstration.

Read more

Visualise Elasticsearch Watcher Statistics with Kibana

My previous post, demonstrated how to use Elasticsearch Watcher for log file alerting. Elasticsearch Watcher itself keeps data, about its watches and actions.

Read more

Alerting with Elasticsearch Watcher

Watcher is a commercial plugin for alerting based on elasticsearch documents. The required knowledge could be overwhelming, but is rather straightforward and pretty simple after understanding the fundamental concepts. This post will give you a simple watch definition to grasp the concept. If you have application logs and store them into elasticsearch, you want to be alerted if a log entry with log level ERROR is reported. Let’s do this.

Read more