Loading...

HTTPS monitoring with Heartbeat

Heartbeat is still beta, but is worth a try. If you have an external REST endpoint and you need a history to check if the endpoint is available, heartbeat is one eligible solution.

Versions:

  • Elasticsearch 5.5.2
  • Kibana 5.5.2
  • Heartbeat 5.5.2

Configuration

First, let’s define the endpoint in the heartbeat.yml

heartbeat.monitors:
- type: http

  urls: ["https://monitoring-test.cinhtau.net","https://monitoring-prod.cinhtau.net"]
  schedule: '@every 60s'
  timeout: 2m
  ssl:
    certificate_authorities: ['/home/tan/ssl/ca.crt']
    supported_protocols: ["TLSv1.2"]
  check.request:
    method: GET
    headers:
      'Authorization': 'Basic bWFwcGVyOmtpbmc='
  check.response:
    status: 200

Monitor Endpoints

The urls field contains all the http endpoints.

urls: ["https://monitoring-test.cinhtau.net","https://monitoring-prod.cinhtau.net"] 

TLS

Since the endpoint is https you have to omit the TLS information. In my case I needed to add the issuer certificate authorities. In my case is Symantec. The certificates are available on their support site.

Just concatenate all certificates into one ca.crt file. Without the information, you will get a X509 certificate error → unknown certificate authority.

ssl:
  certificate_authorities: ['/home/tan/ssl/ca.crt']
  supported_protocols: ["TLSv1.2"]

Security

Since Elasticsearch is protected with basic authentication, I add the auth header to the check request.

check.request:
  method: GET
  headers:
    'Authorization': 'Basic bWFwcGVyOmtpbmc='

Heartbeat checks for the HTTP response code 200 (OK). We could also check for the response body, but since it is subject to change on every elasticsearch upgrade, checking the response code is sufficient.

check.response:
  status: 200

TCP Monitoring

To demonstrate TCP Monitoring, following config checks if logstash has started the beats input plugin on port 5044.

- type: tcp
  schedule: '@every 1m'
  hosts: ["localhost:5044"]  # default TCP Echo Protocol

Additional Information

To add custom fields or custom values in the tags field add them in the General section.

#================================ General =====================================

name: "le-mapper"
tags: ["mapper-king", "web-tier"]
fields:
  env: staging

Reporting Output

The data might be send to logstash or directly to elasticsearch.

#================================ Outputs =====================================

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  username: "elastic"
  password: "secret"

Logging Output

Use the logging section to define the internal output for debugging.

#================================ Logging =====================================

logging.level: info
logging.to_files: true
logging.to_syslog: false
logging.files:
  path: /var/log/beats
  name: heart-beat.log
  keepfiles: 7

A regular output:

2017-09-04T11:36:14+02:00 INFO Setup Beat: heartbeat; Version: 5.5.2
2017-09-04T11:36:14+02:00 INFO Loading template enabled. Reading template file: /home/tan/heartbeat-5.5.2-linux-x86_64/heartbeat.template.json
2017-09-04T11:36:14+02:00 INFO Loading template enabled for Elasticsearch 2.x. Reading template file: /home/tan/heartbeat-5.5.2-linux-x86_64/heartbeat.template-es2x.json
2017-09-04T11:36:14+02:00 INFO Loading template enabled for Elasticsearch 6.x. Reading template file: /home/tan/heartbeat-5.5.2-linux-x86_64/heartbeat.template-es6x.json
2017-09-04T11:36:14+02:00 INFO Elasticsearch url: http://localhost:9200
2017-09-04T11:36:14+02:00 INFO Activated elasticsearch as output plugin.
2017-09-04T11:36:14+02:00 INFO Publisher name: le-mapper
2017-09-04T11:36:14+02:00 INFO Flush Interval set to: 1s
2017-09-04T11:36:14+02:00 INFO Max Bulk Size set to: 50
2017-09-04T11:36:14+02:00 WARN Beta: Heartbeat is beta software
2017-09-04T11:36:14+02:00 INFO Select (active) monitor http
2017-09-04T11:36:14+02:00 INFO Select (active) monitor tcp
2017-09-04T11:36:14+02:00 INFO heartbeat start running.
2017-09-04T11:36:14+02:00 INFO heartbeat is running! Hit CTRL-C to stop it.
2017-09-04T11:36:44+02:00 INFO No non-zero metrics in the last 30s
2017-09-04T11:37:14+02:00 INFO No non-zero metrics in the last 30s
2017-09-04T11:37:15+02:00 INFO Connected to Elasticsearch version 5.5.2
2017-09-04T11:37:15+02:00 INFO Trying to load template for client: http://localhost:9200
2017-09-04T11:37:15+02:00 INFO Template already exists and will not be overwritten.
2017-09-04T11:37:44+02:00 INFO Non-zero metrics in the last 30s: libbeat.es.call_count.PublishEvents=1 libbeat.es.publish.read_bytes=972 libbeat.es.publish.write_bytes=2374 libbeat.es.published_and_acked_events=3 libbeat.publisher.messages_in_worker_queues=3 libbeat.publisher.published_events=3

Data in Elasticsearch

Heartbeat will write this kind of data.

{
  "_index": "heartbeat-2017.09.04",
  "_type": "doc",
  "_id": "AV5MQFLFT-rF7Tttya86",
  "_score": 1,
  "_source": {
    "@timestamp": "2017-09-04T09:37:14.247Z",
    "beat": {
      "hostname": "omega",
      "name": "le-mapper",
      "version": "5.5.2"
    },
    "duration": {
      "us": 155771
    },
    "fields": {
      "env": "staging"
    },
    "host": "monitoring.cinhtau.six-group.net",
    "http_rtt": {
      "us": 36136
    },
    "ip": "10.22.12.118",
    "monitor": "http@https://monitoring.cinhtau.six-group.net",
    "port": 443,
    "resolve_rtt": {
      "us": 60807
    },
    "response": {
      "status": 200
    },
    "rtt": {
      "us": 94785
    },
    "scheme": "https",
    "tags": [
      "mapper-king",
      "web-tier"
    ],
    "tcp_connect_rtt": {
      "us": 10313
    },
    "tls_handshake_rtt": {
      "us": 47684
    },
    "type": "http",
    "up": true,
    "url": "https://monitoring.cinhtau.six-group.net"
  }
}

The Kibana Dashboard

A preset dashboard is shipped within heartbeat.

Heartbeat Dashboard