The new Kibana 5.1.2 x-pack monitoring plugin is a job well done! With the major version, you can also monitor your Kibana instances - yes Kibana can be run clustered . Having Kibana running in Docker, it allows you deploy it even faster. Back to the monitoring part. It was a real lifesaver and gave me some insights on the Kibana application life-cycle.
I notice the amount of http connections is astonishing.
Even I know that our Kibana is popular, but it is very unlikely that this many users and sessions are alive. I just made a control check by restarting one instance. And my suspicion was right. After the docker restart the connection started at zero.
If I compare the instances, I can easily figure out which instance was rebooted and which one runs for days.
My personal suspicion: The DNS network load-balancer is doing a probe on the application port 5601 for the health-check. If you have a probe from 30 seconds to 2 minutes, it would explain the numbers. If the load-balancer is responsible for that, I should give him another port for the health-check or probe. For this task I choose
Netcat is a computer networking utility for reading from and writing to network connections using TCP or UDP.
In the Dockerfile we install
netcat and expose port 5602 for the health check.
FROM kibana:5.1.2 # .. do some setup COPY docker-entrypoint.sh / RUN chmod +x /docker-entrypoint.sh # install netcat RUN apt-get update -y && apt-get install netcat -y EXPOSE 5601 5602 ENTRYPOINT ["/docker-entrypoint.sh"] CMD ["kibana"]
docker-entrypoint.sh just start netcat to listen to port 5602 and reply with hello. The other stuff is for stopping the loop if CTRL + C is pressed, which is in a docker terminal not the usual case. netcat will terminate the connection for sure after echoing hello! ( Adele greets ya )
while [ 1 ]; do nc -l -p 5602 -c "echo hello"; test $? -gt 128 && break; done &
Build and deployed the new docker container for Kibana with integrated health-check port. I talk to the network guys and they change the health-check probe to 5602. Of course it took some time for them to switch it. As you can see the http connections stopped to increase exponentially from the point on they performed the switch.
After a rolling restart I have the regular numbers what I would have expected from the beginning.
From my personal view, I was afraid that Kibana was only an UI improvement, but looking at the commercial plug-ins, it has improved a lot. Sadly they don’t come for free, but that doesn’t mean they aren’t worth their license.
This is only a temporary solution. Use Nginx for instance with Kibana to provide a healthcheck, but ensure to close the connection by the load balancer probe.