1. 2017-08-28 - HTTPS monitoring with Heartbeat; Tags: HTTPS monitoring with Heartbeat
    Loading...

    HTTPS monitoring with Heartbeat

    Heartbeat is still beta, but is worth a try. If you have an external REST endpoint and you need a history to check if the endpoint is available, heartbeat is one eligible solution.

    Configuration

    First, let’s define the endpoint in the heartbeat.yml

    heartbeat.monitors:
    - type: http
    
      urls: ["https://monitoring-test.cinhtau.net","https://monitoring-prod.cinhtau.net"]
      schedule: '@every 60s'
      timeout: 2m
      ssl:
        certificate_authorities: ['/home/tan/ssl/ca.crt']
        supported_protocols: ["TLSv1.2"]
      check.request:
        method: GET
        headers:
          'Authorization': 'Basic bWFwcGVyOmtpbmc='
      check.response:
        status: 200
    

    Monitor Endpoints

    The urls field contains all the http endpoints.

    urls: ["https://monitoring-test.cinhtau.net","https://monitoring-prod.cinhtau.net"] 
    

    TLS

    Since the endpoint is https you have to omit the TLS information. In my case I needed to add the issuer certificate authorities. In my case is Symantec. The certificates are available on their support site.

    Just concatenate all certificates into one ca.crt file. Without the information, you will get a X509 certificate error → unknown certificate authority.

    ssl:
      certificate_authorities: ['/home/tan/ssl/ca.crt']
      supported_protocols: ["TLSv1.2"]
    

    Security

    Since Elasticsearch is protected with basic authentication, I add the auth header to the check request.

    check.request:
      method: GET
      headers:
        'Authorization': 'Basic bWFwcGVyOmtpbmc='
    

    Heartbeat checks for the HTTP response code 200 (OK). We could also check for the response body, but since it is subject to change on every elasticsearch upgrade, checking the response code is sufficient.

    check.response:
      status: 200
    

    TCP Monitoring

    To demonstrate TCP Monitoring, following config checks if logstash has started the beats input plugin on port 5044.

    - type: tcp
      schedule: '@every 1m'
      hosts: ["localhost:5044"]  # default TCP Echo Protocol
    

    Additional Information

    To add custom fields or custom values in the tags field add them in the General section.

    #================================ General =====================================
    
    name: "le-mapper"
    tags: ["mapper-king", "web-tier"]
    fields:
      env: staging
    

    Reporting Output

    The data might be send to logstash or directly to elasticsearch.

    #================================ Outputs =====================================
    
    output.elasticsearch:
      # Array of hosts to connect to.
      hosts: ["localhost:9200"]
    
      # Optional protocol and basic auth credentials.
      #protocol: "https"
      username: "elastic"
      password: "secret"
    

    Logging Output

    Use the logging section to define the internal output for debugging.

    #================================ Logging =====================================
    
    logging.level: info
    logging.to_files: true
    logging.to_syslog: false
    logging.files:
      path: /var/log/beats
      name: heart-beat.log
      keepfiles: 7
    

    A regular output:

    2017-09-04T11:36:14+02:00 INFO Setup Beat: heartbeat; Version: 5.5.2
    2017-09-04T11:36:14+02:00 INFO Loading template enabled. Reading template file: /home/tan/heartbeat-5.5.2-linux-x86_64/heartbeat.template.json
    2017-09-04T11:36:14+02:00 INFO Loading template enabled for Elasticsearch 2.x. Reading template file: /home/tan/heartbeat-5.5.2-linux-x86_64/heartbeat.template-es2x.json
    2017-09-04T11:36:14+02:00 INFO Loading template enabled for Elasticsearch 6.x. Reading template file: /home/tan/heartbeat-5.5.2-linux-x86_64/heartbeat.template-es6x.json
    2017-09-04T11:36:14+02:00 INFO Elasticsearch url: http://localhost:9200
    2017-09-04T11:36:14+02:00 INFO Activated elasticsearch as output plugin.
    2017-09-04T11:36:14+02:00 INFO Publisher name: le-mapper
    2017-09-04T11:36:14+02:00 INFO Flush Interval set to: 1s
    2017-09-04T11:36:14+02:00 INFO Max Bulk Size set to: 50
    2017-09-04T11:36:14+02:00 WARN Beta: Heartbeat is beta software
    2017-09-04T11:36:14+02:00 INFO Select (active) monitor http
    2017-09-04T11:36:14+02:00 INFO Select (active) monitor tcp
    2017-09-04T11:36:14+02:00 INFO heartbeat start running.
    2017-09-04T11:36:14+02:00 INFO heartbeat is running! Hit CTRL-C to stop it.
    2017-09-04T11:36:44+02:00 INFO No non-zero metrics in the last 30s
    2017-09-04T11:37:14+02:00 INFO No non-zero metrics in the last 30s
    2017-09-04T11:37:15+02:00 INFO Connected to Elasticsearch version 5.5.2
    2017-09-04T11:37:15+02:00 INFO Trying to load template for client: http://localhost:9200
    2017-09-04T11:37:15+02:00 INFO Template already exists and will not be overwritten.
    2017-09-04T11:37:44+02:00 INFO Non-zero metrics in the last 30s: libbeat.es.call_count.PublishEvents=1 libbeat.es.publish.read_bytes=972 libbeat.es.publish.write_bytes=2374 libbeat.es.published_and_acked_events=3 libbeat.publisher.messages_in_worker_queues=3 libbeat.publisher.published_events=3
    

    Data in Elasticsearch

    Heartbeat will write this kind of data.

    {
      "_index": "heartbeat-2017.09.04",
      "_type": "doc",
      "_id": "AV5MQFLFT-rF7Tttya86",
      "_score": 1,
      "_source": {
        "@timestamp": "2017-09-04T09:37:14.247Z",
        "beat": {
          "hostname": "omega",
          "name": "le-mapper",
          "version": "5.5.2"
        },
        "duration": {
          "us": 155771
        },
        "fields": {
          "env": "staging"
        },
        "host": "monitoring.cinhtau.six-group.net",
        "http_rtt": {
          "us": 36136
        },
        "ip": "10.22.12.118",
        "monitor": "http@https://monitoring.cinhtau.six-group.net",
        "port": 443,
        "resolve_rtt": {
          "us": 60807
        },
        "response": {
          "status": 200
        },
        "rtt": {
          "us": 94785
        },
        "scheme": "https",
        "tags": [
          "mapper-king",
          "web-tier"
        ],
        "tcp_connect_rtt": {
          "us": 10313
        },
        "tls_handshake_rtt": {
          "us": 47684
        },
        "type": "http",
        "up": true,
        "url": "https://monitoring.cinhtau.six-group.net"
      }
    }
    

    The Kibana Dashboard

    A preset dashboard is shipped within heartbeat.

    Heartbeat Dashboard