Loading...

Monitoring of TCP connections with collectd and Elasticsearch

If you have an application which does distributed computing, i.e. means connects to other servers and send data, it is interesting to monitor the connection handling. Therefore collectd provides the plugin tcpconns, that allows to monitor dedicated ports. This data can be send to logstash, where it can have graphite or elasticsearch as output. Having the data in a metrics storage, visualization with Kibana or Grafana is a piece of cake.

Collectd

Collectd is OpenSource and a daemon which collects system and application performance metrics periodically. The first step is to install collectd.

# deb based install
# apt-get install collectd collectd-utils

# rpm based install
# yum install collectd collectd-utils

Minimal System Configuration

Collectd is very efficient. If we have no any other areas of interest like CPU, Memory, etc. the minimal setup for collectd:

/etc/collectd.conf

LoadPlugin syslog
Include "/etc/collectd.d"

Add your Configuration

The folder /etc/collectd.d should contain our configuration for application and the output destination.

Monitoring Configuration

Example config for tcpconns

LoadPlugin tcpconns
<Plugin "tcpconns">
  ListeningPorts false
  # Outbound connections; i.e. the servers we're connecting to:
  RemotePort "2251"
  RemotePort "3513"
  RemotePort "3504"
  # Locally listening ports; i.e. the servers we're running:
  LocalPort "22"
  LocalPort "25"
  LocalPort "8080"
  LocalPort "8181"
</Plugin>

Output Configuration

collectd writes the collected data to the UDP port 25826 in the binary protocol. As receiving end I chose logstash with its collectd input plugin. Replace IP-address and port to your needs.

out-network.conf

LoadPlugin network
<Plugin network>
    <Server "10.22.12.121" "25826">
    </Server>
</Plugin>

Start collecting data

First we have to enable the service

# systemctl enable collectd.service

Next is to start the service

# systemctl start collectd.service

Check with status if the service is running properly. As you can see the monitoring takes only 508 kb in memory.

root@alpha:/etc/collectd.d# systemctl status collectd.service
● collectd.service - Collectd statistics daemon
   Loaded: loaded (/usr/lib/systemd/system/collectd.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2016-10-24 13:31:46 CEST; 22min ago
     Docs: man:collectd(1)
           man:collectd.conf(5)
 Main PID: 26116 (collectd)
   Memory: 508.0K
   CGroup: /system.slice/collectd.service
           └─26116 /usr/sbin/collectd
Oct 24 13:31:46 alpha systemd[1]: Starting Collectd statistics daemon...
Oct 24 13:31:46 alpha collectd[26116]: supervised by systemd, will signal readyness
Oct 24 13:31:46 alpha systemd[1]: Started Collectd statistics daemon.
Oct 24 13:31:46 alpha collectd[26116]: Initialization complete, entering read-loop.
Oct 24 13:31:46 alpha collectd[26116]: tcpconns plugin: Reading from netlink succeeded. Will use the netlink method from now on.

Logstash

On the receiving end of collectd is logstash. Logstash needs an input configuration for collectd.

100_input_collectd.conf

input {
  udp {
    port => 25826
    buffer_size => 1452
    codec => collectd { }
  }
}

Logstash can read the collectd data, but must also decide where to store the data. As fitting endpoint I take Elasticsearch.

The collected data shall be written to the metrics index.

900_output_es.conf

output {
if [collectd_type] =~ /.+/ {
	elasticsearch {
		hosts => [ "es-1", "es-2", "es-aws" ]
		index => "metrics-%{+xxxx.ww}"
	}
}}

Notice that the if condition is not necessary, I only ensure to write all events that contains the field collectd_type to the metrics index.

Since the data is in Elasticsearch we can visualize the data in Kibana, see the example visualization for SSH. This can be embedded into existing dashboards.

Summary

collectd is an easy and lightweight method to monitor network connections. If you stay low profile, and do not monitor cpu, memory and many other stuff, you have a simple, brain dead simple and robust monitoring.