1. 2017-10-05 - Ship Monit logs with Filebeat; Tags: Ship Monit logs with Filebeat
    Loading...

    Ship Monit logs with Filebeat

    A quick recipe how to ship Monit logs to Elasticsearch. Some initial configuration was in place but I ran into some troubles.

    Problems

    1. Monit logs disappeared on the 1st of October
    2. Multiline messages haven’t been properly processed

    Example logs

    [CEST Oct  5 16:54:19] error    : 'batch_healthcheck' failed protocol test [HTTP] at [0.0.0.0]:80/batch/systemStatus [TCP/IP] -- Connection refused
    [CEST Oct  5 16:54:19] info     : 'batch_healthcheck' exec: /bin/bash
    [CEST Oct  5 16:54:19] error    : 'imp4' process is not running
    [CEST Oct  5 16:54:19] info     : 'imp4' trying to restart
    [CEST Oct  5 16:54:19] info     : 'imp4' start: /opt/six/fo/jboss/bin/jboss-opr.sh
    [CEST Oct  5 16:54:52] error    : 'imp4' failed to start (exit status 1) -- /opt/six/fo/jboss/bin/jboss-opr.sh: tput: No value for $TERM and no -T specified
    tput: No value for $TERM and no -T specified
    2017-10-05 16:54:19 root jboss.sh: imp4 not known
    2017-10-05 16:54:19 root jboss.sh: imp4 could not be started
    

    1. Monit logs disappeared

    The reason is a simple one. Monit just don’t log day of month with leading zero.

    [CEST Oct  5 16:54:19] error    : 'imp4' process is not running
    

    Just add the date pattern for a single day in the date processor formats.

    {
      "date": {
        "field": "jesus",
        "target_field": "datetime",
        "formats": [
          "MMM dd HH:mm:ss",
          "MMM  d HH:mm:ss"
        ],
        "timezone": "Europe/Zurich"
      }
    }
    

    2. Multiline messages

    Filebeat sends multiline messages by this configuration:

    multiline.pattern: '^\['
    multiline.negate: false
    multiline.match: before
    

    In the ingest pipeline the message is truncated since the newline character collides with the grok processor. With the gsub processor you can remove the newline in order to be properly kept by the grok processor.

    {
      "gsub": {
        "field": "message",
        "pattern": "\n",
        "replacement": " "
      }
    }
    

    Solution

    Extend pipeline

    PUT _ingest/pipeline/monit_logs

    {
      "description": "grok pipeline for monit logs",
      "processors": [
        {
          "gsub": {
            "field": "message",
            "pattern": "\n",
            "replacement": " "
          }
        },
        {
          "grok": {
            "field": "message",
            "patterns": [
              ""
              "\[%{WORD} %{GREEDYDATA:jesus}\] %{WORD:level} %{SPACE} : \'%{GREEDYDATA:service}\' %{GREEDYDATA:logmessage}"
              ""
            ]
          }
        },
        {
          "date": {
            "field": "jesus",
            "target_field": "datetime",
            "formats": [
              "MMM dd HH:mm:ss",
              "MMM  d HH:mm:ss"
            ],
            "timezone": "Europe/Zurich"
          }
        },
        {
          "remove": {
            "field": [
              "message",
              "jesus"
            ]
          }
        }
      ],
      "on_failure": [
        {
          "set": {
            "field": "error",
            "value": " on operation: "
          }
        }
      ]
    }
    

    Configure filebeat

    # monit logs
    - input_type: log
      paths:
         - /var/log/monit.log
      exclude_files: [".gz$"]
      fields:
        type: "logs"
        host: "${beat.hostname:dhost}"
        application: "monit"
        environment: "${FO_ENV:default}"
      fields_under_root: true
      multiline.pattern: '^\['
      multiline.negate: false
      multiline.match: before
      pipeline: "monit_logs"
    
  2. 2017-07-13 - Service Dependencies in Monit; Tags: Service Dependencies in Monit
    Loading...

    Service Dependencies in Monit

    Monit allows start, stop and restart program instructions in process checks. If you restart with monit, no alarm or warning is triggered then. For instance you need to restart an application like logstash for configuration changes.

    We have for example this control file

    >CHECK PROCESS logstash WITH MATCHING "logstash/runner.rb -f /etc/logstash\s" start program = "/usr/bin/sudo /opt/logstash start" as uid "logstash" and gid "logstash" stop program = "/usr/bin/sudo /opt/logstash stop" as uid "logstash" and gid "logstash" if failed port 5400 type tcpssl then alert alert mapper-king@cinhtau.net but not on { action, instance } GROUP logstash

    To restart the application without monit alert. If you restart logstash otherwise, it will generate an alert, since monit monitors it. :wink:

    sudo monit restart logstash 
    

    You can access the start and stop instructions in other sections, if they depend on the process. For instance stop logstash if the condition is given. You can also use restart, which will just use stop and start.

    CHECK PROCESS logstash WITH MATCHING "logstash/runner.rb -f /etc/logstash\s"
      start program = "/usr/bin/sudo /opt/logstash start" as uid "logstash" and gid "logstash"
      stop program = "/usr/bin/sudo /opt/logstash stop" as uid "logstash" and gid "logstash"
      depends on logstash_logfile
      if failed port 5400 type tcpssl then alert
          alert mapper-king@cinhtau.net but not on { action, instance }
      GROUP logstash
    
    CHECK FILE logstash_logfile with path /var/log/logstash/logstash-plain.log  
      if match "IllegalArgumentException" for 5 times within 5 cycles then stop    
      GROUP logstash
    

    For more information look at the official manual.

  3. 2016-11-08 - Monitor Elasticsearch in Docker with Monit; Tags: Monitor Elasticsearch in Docker with Monit
    Loading...

    Monitor Elasticsearch in Docker with Monit

    Running Elasticsearch as docker container is straightforward. If you don’t have a cluster manager like Kubernetes, monit can help you to keep track of the container lifecycle.

    An exemplary monit configuration:

    CHECK PROCESS elasticsearch WITH MATCHING "org.elasticsearch.bootstrap.Elasticsearch"
    CHECK PROGRAM elasticsearch_container WITH PATH "/usr/bin/docker top elasticsearch"
      if status != 0 then alert
        alert warning@cinhtau.net
      group elkstack
    CHECK HOST elasticsearch_healthcheck WITH ADDRESS cinhtau.net
      if failed url http://cinhtau.net:9200 for 5 cycles
        then alert
          alert warning@cinhtau.net BUT not on { action, instance }
      depends on elasticsearch_container
      group elkstack
    CHECK FILE elasticsearch_logfile with path /var/log/elasticsearch/test-cluster.log
      if match "ERROR" for 2 times within 5 cycles then alert
        alert elasticsearch@cinhtau.net BUT not on { action, instance, nonexist }
      depends on elasticsearch_container
      group elkstack
    

    Pay attention to the nonexist option. Monit does an implicit check if the logifle exists. Elasticsearch writes a log file. Our housekeeping, logrotate or some kind of janitor script, rename, compress or delete this file. So if the file is missing, monit would complain without the option. If the file doesn’t exists, which is basically good for prod, you don’t want to be notified or warned. No logs, no errors, no worries.