1. 2017-01-16 - Provide health check port for Kibana; Tags: Provide health check port for Kibana
    Loading...

    Provide health check port for Kibana

    The new Kibana 5.1.2 x-pack monitoring plugin is a job well done! With the major version, you can also monitor your Kibana instances - yes Kibana can be run clustered :sunglasses:. Having Kibana running in Docker, it allows you deploy it even faster. Back to the monitoring part. It was a real lifesaver and gave me some insights on the Kibana application life-cycle.

    I notice the amount of http connections is astonishing. :dizzy_face:

    Connection Problems

    Even I know that our Kibana is popular, but it is very unlikely that this many users and sessions are alive. I just made a control check by restarting one instance. And my suspicion was right. After the docker restart the connection started at zero.

    Connections after restart

    If I compare the instances, I can easily figure out which instance was rebooted and which one runs for days. :wink:

    Memory Consumption

    My personal suspicion: The DNS network load-balancer is doing a probe on the application port 5601 for the health-check. If you have a probe from 30 seconds to 2 minutes, it would explain the numbers. If the load-balancer is responsible for that, I should give him another port for the health-check or probe. For this task I choose netcat.

    Netcat is a computer networking utility for reading from and writing to network connections using TCP or UDP.

    Netcat Homepage

    In the Dockerfile we install netcat and expose port 5602 for the health check.

    >FROM kibana:5.1.2 # .. do some setup COPY docker-entrypoint.sh / RUN chmod +x /docker-entrypoint.sh # install netcat RUN apt-get update -y && apt-get install netcat -y EXPOSE 5601 5602 ENTRYPOINT ["/docker-entrypoint.sh"] CMD ["kibana"]

    In docker-entrypoint.sh just start netcat to listen to port 5602 and reply with hello. The other stuff is for stopping the loop if CTRL + C is pressed, which is in a docker terminal not the usual case. netcat will terminate the connection for sure after echoing hello! ( Adele greets ya :wink: )

    while [ 1 ]; do nc -l -p 5602 -c "echo hello"; test $? -gt 128 && break; done &
    

    Build and deployed the new docker container for Kibana with integrated health-check port. I talk to the network guys and they change the health-check probe to 5602. Of course it took some time for them to switch it. As you can see the http connections stopped to increase exponentially from the point on they performed the switch.

    Increase Stop

    After a rolling restart I have the regular numbers what I would have expected from the beginning.

    Normal Connections

    From my personal view, I was afraid that Kibana was only an UI improvement, but looking at the commercial plug-ins, it has improved a lot. Sadly they don’t come for free, but that doesn’t mean they aren’t worth their license.

    This is only a temporary solution. Use Nginx for instance with Kibana to provide a healthcheck, but ensure to close the connection by the load balancer probe.

  2. 2016-10-25 - Monitoring of TCP connections with collectd and Elasticsearch; Tags: Monitoring of TCP connections with collectd and Elasticsearch
    Loading...

    Monitoring of TCP connections with collectd and Elasticsearch

    If you have an application which does distributed computing, i.e. means connects to other servers and send data, it is interesting to monitor the connection handling. Therefore collectd provides the plugin tcpconns, that allows to monitor dedicated ports. This data can be send to logstash, where it can have graphite or elasticsearch as output. Having the data in a metrics storage, visualization with Kibana or Grafana is a piece of cake.

    Collectd

    Collectd is OpenSource and a daemon which collects system and application performance metrics periodically. The first step is to install collectd.

    # deb based install
    # apt-get install collectd collectd-utils
    
    # rpm based install
    # yum install collectd collectd-utils
    

    Minimal System Configuration

    Collectd is very efficient. If we have no any other areas of interest like CPU, Memory, etc. the minimal setup for collectd:

    /etc/collectd.conf

    LoadPlugin syslog
    Include "/etc/collectd.d"
    

    Add your Configuration

    The folder /etc/collectd.d should contain our configuration for application and the output destination.

    Monitoring Configuration

    Example config for tcpconns

    LoadPlugin tcpconns
    <Plugin "tcpconns">
      ListeningPorts false
      # Outbound connections; i.e. the servers we're connecting to:
      RemotePort "2251"
      RemotePort "3513"
      RemotePort "3504"
      # Locally listening ports; i.e. the servers we're running:
      LocalPort "22"
      LocalPort "25"
      LocalPort "8080"
      LocalPort "8181"
    </Plugin>
    

    Output Configuration

    collectd writes the collected data to the UDP port 25826 in the binary protocol. As receiving end I chose logstash with its collectd input plugin. Replace IP-address and port to your needs.

    out-network.conf

    LoadPlugin network
    <Plugin network>
        <Server "10.22.12.121" "25826">
        </Server>
    </Plugin>
    

    Start collecting data

    First we have to enable the service

    # systemctl enable collectd.service
    

    Next is to start the service

    # systemctl start collectd.service
    

    Check with status if the service is running properly. As you can see the monitoring takes only 508 kb in memory.

    root@alpha:/etc/collectd.d# systemctl status collectd.service
    ● collectd.service - Collectd statistics daemon
       Loaded: loaded (/usr/lib/systemd/system/collectd.service; enabled; vendor preset: disabled)
       Active: active (running) since Mon 2016-10-24 13:31:46 CEST; 22min ago
         Docs: man:collectd(1)
               man:collectd.conf(5)
     Main PID: 26116 (collectd)
       Memory: 508.0K
       CGroup: /system.slice/collectd.service
               └─26116 /usr/sbin/collectd
    Oct 24 13:31:46 alpha systemd[1]: Starting Collectd statistics daemon...
    Oct 24 13:31:46 alpha collectd[26116]: supervised by systemd, will signal readyness
    Oct 24 13:31:46 alpha systemd[1]: Started Collectd statistics daemon.
    Oct 24 13:31:46 alpha collectd[26116]: Initialization complete, entering read-loop.
    Oct 24 13:31:46 alpha collectd[26116]: tcpconns plugin: Reading from netlink succeeded. Will use the netlink method from now on.
    

    Logstash

    On the receiving end of collectd is logstash. Logstash needs an input configuration for collectd.

    100_input_collectd.conf

    input {
      udp {
        port => 25826
        buffer_size => 1452
        codec => collectd { }
      }
    }
    

    Logstash can read the collectd data, but must also decide where to store the data. As fitting endpoint I take Elasticsearch.

    The collected data shall be written to the metrics index.

    900_output_es.conf

    output {
    if [collectd_type] =~ /.+/ {
    	elasticsearch {
    		hosts => [ "es-1", "es-2", "es-aws" ]
    		index => "metrics-%{+xxxx.ww}"
    	}
    }}
    

    Notice that the if condition is not necessary, I only ensure to write all events that contains the field collectd_type to the metrics index.

    Since the data is in Elasticsearch we can visualize the data in Kibana, see the example visualization for SSH. This can be embedded into existing dashboards.

    Summary

    collectd is an easy and lightweight method to monitor network connections. If you stay low profile, and do not monitor cpu, memory and many other stuff, you have a simple, brain dead simple and robust monitoring.