A Journey of a Thousand Miles Begins with a Single Step

Monitor Kibana queries with Packetbeat

If you are using X-Pack Monitoring you have a good overview of your Kibana performance. Sometimes it is necessary to know more. Packetbeat can monitor your http traffic between Kibana and the Elasticsearch node.

Read more

Dashboard with id x not found

X-Pack Reporting allows to automate and generate daily reports on pre-existing dashboards or visualizations in Kibana. To keep security tight I have created a reporting user. The first run with the reporting user gave me some mystery. Reporting complained Dashboard with id 'AWLOnWVZLaWygeBEGxLJ' not found. I did some digging and found the reason, which I am going to elaborate about in this post.

Read more

Using Proxy for Python on Windows

Using Python on Windows is not my first choice but if you have to, here are some recipes how to use pip behind a proxy. This post assumes that you are using CNTLM as running proxy.

Read more

Check active users

top gives you information about active users on a linux server system.

Read more

Watch Zombie Processes on Linux

On Unix and Unix-like computer operating systems, a zombie process or defunct process is a process that has completed execution (via the exit system call) but still has an entry in the process table: it is a process in the “Terminated state”.

Read more

HTTP Input for Elasticsearch Watcher

Elasticsearch X-Pack Alerting or aka Watcher offers the capability to alert on specific events/constellation in the Elasticsearch data. Watcher can retrieve data from the cluster where it runs (on the master node), or fetch data from Restful Web-Services via the http input. Preferably having a production cluster, you should report the monitoring data to a dedicated Elasticsearch monitoring cluster. This monitoring cluster can also run watches. The watch I’am going to introduce is the cluster health watch.

Read more

Shard Allocation in a Elasticsearch Cluster

Shards are parts of an Apache Lucene Index, the storage unit of Elasticsearch. An index may consists of more than one shard. Elasticsearch distributes the storage to its nodes. In a regular case each shard (as primary) has a replica. Primary and Replica are never stored on the same node. If a node fails, the replica takes over as primary and Elasticsearch tries to allocate a replica shard in the remaining cluster nodes. Cluster Shard Allocation is a pretty decent mechanism to ensure high availability. This post gives some insights and recipes how to deal with cluster shard allocation in a hot-warm architecture.

Read more

Elasticsearch Certificates

Since Version 6 X-Pack Security for Elasticsearch requires Node to Node encryption to secure the Elasticsearch cluster. The main reason is, that no unknown node can join the cluster and gets data by shard allocation. Since V6, V6.1 and V6.2 the tool certgen became deprecated and was replaced by certutil. My use case scenario: Created certificates with certgen for my cluster and needed to generate a new certificate for a new data node.

Read more

Using Sidecar Container for Elasticsearch Configuration

Applications shipped in Docker containers are a major game changer, especially having a Elasticsearch cluster. My production cluster consists of 11 nodes. In the core, Elasticsearch is the same. Each node though has its specific configuration, settings and purpose. On top of that, Elasticsearch X-Pack Security in Version 6 requires that the communication within the cluster must run encrypted. This is accomplished by SSL certificates. Each node has its own private key and certificate. So I was facing with the problem, how to ship the node specific parts along with the core elasticsearch container. Use the core container as baseline and copy the configuration and certificate into the container? This would resolve in 11 specific images. Not in the spirit of reusability though. :thinking: The better approach or answer came by remembering the tech talk Docker Patterns by Roland Huss, given at the Java Conference (Javaland 2016). Use a configuration container as a sidecar!

Read more