1. 2017-07-15 - Import Currency codes into Elasticsearch; Tags: Import Currency codes into Elasticsearch
    Loading...

    Import Currency codes into Elasticsearch

    Working in the financial business requires to have the currency code master-data accessible for various reasons. The ISO 4217 currency codes can be obtained from the ISO Organization website. This post uses Logstash and the csv plugin to process the data and import it into Elasticsearch. Elasticsearch itself provides the REST interface, so every micro-service or web service can access the desired data.

    Export the data from Excel to CSV. This logstash configuration reads the CSV data, converts it and ships it to elasticsearch. Alter the values to your scenario.

    input {
      file {
        path => "/tmp/currencies.csv"
        start_position => "beginning"
        sincedb_path => "/dev/null"
      }
    }
    filter {
      csv {
        columns => ["entity", "currency", "alphaCode", "id", "numericCode", "minorUnit"]
        separator => ";"
      }
      mutate {
        remove_field => "message"
        convert => {
            "numericCode" => "integer"
            "minorUnit" => "integer"
        }
        add_field => {
            "[meta][edition]" => "ISO 4217:2015"
        }
        replace => {
            "id" => "%{alphaCode}-%{entity}"
        }
      }
    }
    output {
      stdout { codec => "rubydebug" }
      elasticsearch {
        hosts => [ "elasticsearch:9200" ]
        user => "elastic"
        password => "changeme"
        index => "masterdata"
        document_type => "currency"
        document_id => "%{id}"
      }
    }
                 
  2. 2017-07-13 - Service Dependencies in Monit; Tags: Service Dependencies in Monit
    Loading...

    Service Dependencies in Monit

    Monit allows start, stop and restart program instructions in process checks. If you restart with monit, no alarm or warning is triggered then. For instance you need to restart an application like logstash for configuration changes.

    We have for example this control file

    >CHECK PROCESS logstash WITH MATCHING "logstash/runner.rb -f /etc/logstash\s" start program = "/usr/bin/sudo /opt/logstash start" as uid "logstash" and gid "logstash" stop program = "/usr/bin/sudo /opt/logstash stop" as uid "logstash" and gid "logstash" if failed port 5400 type tcpssl then alert alert mapper-king@cinhtau.net but not on { action, instance } GROUP logstash

    To restart the application without monit alert. If you restart logstash otherwise, it will generate an alert, since monit monitors it. :wink:

    sudo monit restart logstash 
    

    You can access the start and stop instructions in other sections, if they depend on the process. For instance stop logstash if the condition is given. You can also use restart, which will just use stop and start.

    CHECK PROCESS logstash WITH MATCHING "logstash/runner.rb -f /etc/logstash\s"
      start program = "/usr/bin/sudo /opt/logstash start" as uid "logstash" and gid "logstash"
      stop program = "/usr/bin/sudo /opt/logstash stop" as uid "logstash" and gid "logstash"
      depends on logstash_logfile
      if failed port 5400 type tcpssl then alert
          alert mapper-king@cinhtau.net but not on { action, instance }
      GROUP logstash
    
    CHECK FILE logstash_logfile with path /var/log/logstash/logstash-plain.log  
      if match "IllegalArgumentException" for 5 times within 5 cycles then stop    
      GROUP logstash
    

    For more information look at the official manual.

  3. 2017-01-24 - Fix timestamp parse failure in Elasticsearch; Tags: Fix timestamp parse failure in Elasticsearch
    Loading...

    Fix timestamp parse failure in Elasticsearch

    If you use logstash to send logs to elasticsearch in JSON format, you may experience a _timestampparsefailure for this date format [24/Jan/2017:09:04:07 +0100]. This date format is often used in the access.log e.g. JBoss EAP. To solve that, just add an additional date format pattern into the index template.

    The date [24/Jan/2017:09:04:07 +0100] has following format [dd/MMM/yyyy:HH:mm:ss Z]. The format is derived from JodaTime. We can add the new pattern to the index template, that defines the data mapping.

    A demonstration: Create index for test.

    PUT testdata
    {
      "settings": {
        "number_of_shards": 1
      },
      "mappings": {
        "_default_" :{
          "properties": {
            "@timestamp": {
              "type":   "date",
              "format": "\[dd/MMM/yyyy:HH:mm:ss Z\]"
            }
          }
        }
      }
    }
    

    Now we create a test entry, and look if Elasticsearch complains.

    POST testdata/logs
    {
      "@timestamp": "[24/Jan/2017:09:04:07 +0100]",
      "message" : "Salut Phillipe"
    }
    

    Now we query the created log entry.

    GET testdata/logs/_search
    {
      "query": { "match_all": {}}
    }
    

    And the result:

    TODO
    
  4. 2016-12-04 - Parse user agent strings into structured data based on BrowserScope data with logstash; Tags: Parse user agent strings into structured data based on BrowserScope data with logstash
    Loading...

    Parse user agent strings into structured data based on BrowserScope data with logstash

    The Apache HTTP Server logs user agent strings. The user agent string contains information like family, operating system, version, and device. Logstash offers a filter plugin to parse this information.

    Following exemplary log entry.

    66.249.76.151 - - [04/Dec/2016:02:14:01 +0100] "GET /wp/docker-behind-proxy-with-cntlm/ HTTP/1.1" 200 14386 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    

    Use in the filter section following configuration for useragent.

    input { stdin { } }
    filter {
      grok {
        match => { "message" => "%{COMBINEDAPACHELOG}" }
      }
      date {
        match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
      }
      geoip {
        source => "clientip"
        target => "geoip"
        database => "./GeoLite2-City.mmdb"
        add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
        add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
      }
      mutate {
        convert => [ "[geoip][coordinates]", "float"]
        add_field => { "dnsname" => "%{clientip}" }
      }
      dns {
        reverse => [ "dnsname" ]
        action => "replace"
      }
      useragent {
        source => "agent"
      }
    }
    output { stdout { codec => "rubydebug" } }
    

    You will receive this output

    {
            "request" => "/wp/docker-behind-proxy-with-cntlm/",
              "agent" => "\"Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)\"",
              "minor" => "1",
               "auth" => "-",
              "ident" => "-",
           "os_minor" => "0",
           "os_major" => "6",
              "major" => "2",
           "clientip" => "66.249.76.151",
           "@version" => "1",
               "host" => "omega",
            "dnsname" => "crawl-66-249-76-151.googlebot.com",
          "timestamp" => "04/Dec/2016:02:14:01 +0100",
              "geoip" => {
                  "timezone" => "America/Los_Angeles",
                        "ip" => "66.249.76.151",
                  "latitude" => 37.419200000000004,
               "coordinates" => [
                [0] -122.0574,
                [1] 37.419200000000004
            ],
            "continent_code" => "NA",
                 "city_name" => "Mountain View",
             "country_code2" => "US",
              "country_name" => "United States",
                  "dma_code" => 807,
             "country_code3" => "US",
               "region_name" => "California",
                  "location" => [
                [0] -122.0574,
                [1] 37.419200000000004
            ],
               "postal_code" => "94043",
                 "longitude" => -122.0574,
               "region_code" => "CA"
        },
                 "os" => "Android 6.0.1",
               "verb" => "GET",
            "message" => "66.249.76.151 - - [04/Dec/2016:02:14:01 +0100] \"GET /wp/docker-behind-proxy-with-cntlm/ HTTP/1.1\" 200 14386 \"-\" \"Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)\"",
           "referrer" => "\"-\"",
         "@timestamp" => 2016-12-04T01:14:01.000Z,
           "response" => "200",
              "bytes" => "14386",
               "name" => "Googlebot",
            "os_name" => "Android",
        "httpversion" => "1.1",
             "device" => "Spider"
    }
    
  5. 2016-10-25 - Monitoring of TCP connections with collectd and Elasticsearch; Tags: Monitoring of TCP connections with collectd and Elasticsearch
    Loading...

    Monitoring of TCP connections with collectd and Elasticsearch

    If you have an application which does distributed computing, i.e. means connects to other servers and send data, it is interesting to monitor the connection handling. Therefore collectd provides the plugin tcpconns, that allows to monitor dedicated ports. This data can be send to logstash, where it can have graphite or elasticsearch as output. Having the data in a metrics storage, visualization with Kibana or Grafana is a piece of cake.

    Collectd

    Collectd is OpenSource and a daemon which collects system and application performance metrics periodically. The first step is to install collectd.

    # deb based install
    # apt-get install collectd collectd-utils
    
    # rpm based install
    # yum install collectd collectd-utils
    

    Minimal System Configuration

    Collectd is very efficient. If we have no any other areas of interest like CPU, Memory, etc. the minimal setup for collectd:

    /etc/collectd.conf

    LoadPlugin syslog
    Include "/etc/collectd.d"
    

    Add your Configuration

    The folder /etc/collectd.d should contain our configuration for application and the output destination.

    Monitoring Configuration

    Example config for tcpconns

    LoadPlugin tcpconns
    <Plugin "tcpconns">
      ListeningPorts false
      # Outbound connections; i.e. the servers we're connecting to:
      RemotePort "2251"
      RemotePort "3513"
      RemotePort "3504"
      # Locally listening ports; i.e. the servers we're running:
      LocalPort "22"
      LocalPort "25"
      LocalPort "8080"
      LocalPort "8181"
    </Plugin>
    

    Output Configuration

    collectd writes the collected data to the UDP port 25826 in the binary protocol. As receiving end I chose logstash with its collectd input plugin. Replace IP-address and port to your needs.

    out-network.conf

    LoadPlugin network
    <Plugin network>
        <Server "10.22.12.121" "25826">
        </Server>
    </Plugin>
    

    Start collecting data

    First we have to enable the service

    # systemctl enable collectd.service
    

    Next is to start the service

    # systemctl start collectd.service
    

    Check with status if the service is running properly. As you can see the monitoring takes only 508 kb in memory.

    root@alpha:/etc/collectd.d# systemctl status collectd.service
    ● collectd.service - Collectd statistics daemon
       Loaded: loaded (/usr/lib/systemd/system/collectd.service; enabled; vendor preset: disabled)
       Active: active (running) since Mon 2016-10-24 13:31:46 CEST; 22min ago
         Docs: man:collectd(1)
               man:collectd.conf(5)
     Main PID: 26116 (collectd)
       Memory: 508.0K
       CGroup: /system.slice/collectd.service
               └─26116 /usr/sbin/collectd
    Oct 24 13:31:46 alpha systemd[1]: Starting Collectd statistics daemon...
    Oct 24 13:31:46 alpha collectd[26116]: supervised by systemd, will signal readyness
    Oct 24 13:31:46 alpha systemd[1]: Started Collectd statistics daemon.
    Oct 24 13:31:46 alpha collectd[26116]: Initialization complete, entering read-loop.
    Oct 24 13:31:46 alpha collectd[26116]: tcpconns plugin: Reading from netlink succeeded. Will use the netlink method from now on.
    

    Logstash

    On the receiving end of collectd is logstash. Logstash needs an input configuration for collectd.

    100_input_collectd.conf

    input {
      udp {
        port => 25826
        buffer_size => 1452
        codec => collectd { }
      }
    }
    

    Logstash can read the collectd data, but must also decide where to store the data. As fitting endpoint I take Elasticsearch.

    The collected data shall be written to the metrics index.

    900_output_es.conf

    output {
    if [collectd_type] =~ /.+/ {
    	elasticsearch {
    		hosts => [ "es-1", "es-2", "es-aws" ]
    		index => "metrics-%{+xxxx.ww}"
    	}
    }}
    

    Notice that the if condition is not necessary, I only ensure to write all events that contains the field collectd_type to the metrics index.

    Since the data is in Elasticsearch we can visualize the data in Kibana, see the example visualization for SSH. This can be embedded into existing dashboards.

    Summary

    collectd is an easy and lightweight method to monitor network connections. If you stay low profile, and do not monitor cpu, memory and many other stuff, you have a simple, brain dead simple and robust monitoring.

  6. 2016-09-15 - Handling logstash input multiline codec; Tags: Handling logstash input multiline codec
    Loading...

    Handling logstash input multiline codec

    The multiline codec will collapse multiline messages and merge them into a single event. The default limit is 500 lines. If there are over 500 lines appended, the multiline codec split the message to the next 500 lines and so forth. This post demonstrates how to deal with this situation. Elasticsearch receives in tags the multiline_codec_max_lines_reached.

        "tags" => [
            [0] "multiline",
            [1] "multiline_codec_max_lines_reached"
        ]
    

    Basically you can use a regular expression to handle these lines. As testing basis I just take some java application logs, e.g. JBoss EAP.

    2016-09-14 16:47:12,845 INFO  [default-threads - 22] [497a1672-52ff-42b7-b53c-df202834c2f5] [APP] [] [] [TERMINAL] (TxManager) Final response message with escaped sensitive data: ...
    

    Let’s assume a regular log line always starts with an ISO Timestamp. For demonstration purposes I lower the regular limit to 25 lines.

    input {
      file {
        path => "/var/log/test/server.log"
        start_position => beginning
        codec => multiline {
          pattern => "^%{TIMESTAMP_ISO8601}"
          negate => true
          what => previous
    	  max_lines => 25
        }
        sincedb_path => "/dev/null"
      }
    }
    output { stdout => { codec => "rubydebug" }}
    

    You can use a regular expression (line starts with ISO timestamp) to properly grok the message and in else do anything you like, e.g. drop the message.

    filter {
      if [message] =~ "^([0-9]{4})-?(1[0-2]|0[1-9])-?(3[01]|0[1-9]|[12][0-9])" {
    	grok {
    	  match => { "message" => "%{TIMESTAMP_ISO8601:datetime}\s%{WORD:level}\s*\[%{DATA:thread}\]\s\[%{DATA:requestid}?\]\s+\[%{WORD:appid}?\]\s+\[%{DATA:sessionid}?\]\s+\[%{DATA:trxid}?\]\s+\[%{DATA:terminalid}?\]\s+\(%{DATA:class}\)\s+%{GREEDYDATA:logmessage}?" }
    	}
      }
      else {
    	# drop continued multilines
    	drop { }
      }
    }
    

    If you do nothing just ensure not to grok the message or grok it properly.

  7. 2016-09-08 - Migrate elasticsearch indices from different clusters with logstash; Tags: Migrate elasticsearch indices from different clusters with logstash
    Loading...

    Migrate elasticsearch indices from different clusters with logstash

    I got an exceptional case in the office. Some application logs, which belongs to a dev and testing environment, were stored or reported in the elasticsearch production cluster. Therefore a cleanup or migration was necessary.

    logstash is an easy solution for migrating data from cluster a to cluster b. In my case cluster production to cluster test. logstash provides elasticsearch as input and output plugin. The input queries elasticsearch and retieves the documents as json. The output writes the json to the target elasticsearch cluster. An example configuration.

    vinh@omega:~/logstash-2.4.0> cat copy-data.conf
    input{
        elasticsearch {
            hosts => [ "prod-dc1", "prod-dc2", "alpha-dc2", "beta-dc2" ]
            index => "trx-*"
            user => "admin"
            password => "SiriSingsRiri"
        }
    }
    output {
        stdout { codec => "rubydebug" }
        elasticsearch {
            hosts => [ "dev", "delta", "gamma" ]
            index => "trx-%{+YYYY.MM.dd}"
        }
    }
    
  8. 2016-06-29 - Resolve multiline grokparsefailure with regex; Tags: Resolve multiline grokparsefailure with regex
    Loading...

    Resolve multiline grokparsefailure with regex

    On a daily logrotate I have noticed that one of my monitored applications is doing a multiline input, but due to a misconfiguration in logstash, the grokparsefailure appears. I just want to leave the message as it is. Therefore logstash provides event dependent configuration capabilities with regular expression comparisons.

    Following Kibana picture shows the message and its grokparsefailure in the field tags.

    grok parse failure

    Every daily logrotate starts with three dots (...). A small extension in the filter configuration does the work:

    filter {
        # if not daily log rotate
        if [message] !~ "^(\.{3})" {
            grok {
              # do some grokking
            }
        }
    }
    

    !~ is a regex expression that checks, that the message does not start with three dots. Since a dot is a special sign in regex, it needs an escape and we use the quantifier 3 to omit the occurence of three dots.

    Regular expressions can be hard, therefore I recommend using for logstash (ruby) a regex tester like Rubular (see below) for testing.

    For more information about Logstash Event Dependent Configuration.

  9. 2016-06-26 - Replace NaN values with zero; Tags: Replace NaN values with zero
    Loading...

    Replace NaN values with zero

    In logstash some grokked fields in messages, that suppose to be numeric, flying in with “NaN” (Not a Number). You can convert them with the mutate filter plugin.

    if [value] == "NaN" {
        mutate {
            replace => { "value" => 0 }
        }
    }
    
  10. 2016-03-30 - Logging from HP NonStop to Elasticsearch cluster; Tags: Logging from HP NonStop to Elasticsearch cluster
    Loading...

    Logging from HP NonStop to Elasticsearch cluster

    This article demonstrates the fundemental milestones to get a decent log reporting on the HP NonStop to an Elasticsearch cluster. The HP NonStop itself offers with OSS an minimal Linux OS on top of the Guardian layer. Following articles involves the configuration on the HP NonStop (sending party) to the Linux Server, that runs Logstash and Elasticsearch (receiving party). We will also call the HP NonStop Tandem, for clarification.

    The scenario

    This article needs a basic understanding of Logstash and HP NonStop OSS. The circumstances are: My company has a HP NonStop (Itanium architecture). On the Tandem machine, several tomcat web applications are running and logging. Viewing the log files with tail under OSS is a pain in the .. you know where :wink: . So the basic idea is to report the log files to elasticsearch and view them with Kibana. The HP NonStop isn’t capable of running logstash (problems with JRuby), logstash-forwarder or filebeat (written in Go). There is an unofficial logstash forwarder implmentation in github. This programme was written for the IBM AIX and fits the purpose of running basic java applications on the Itanium architecture.

    Getting started

    Before we may begin we need to create self signed SSL certificates, that are essential for the logstash forwarder protocol lumberjack and the logstash input configuration. Logstash supports all certificates, including self-signed certificates. To generate a certificate, we run the following command on the Linux Server (receiving party):

    >$ openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout logstash-forwarder.key -out logstash-forwarder.crt -days 365

    This will generate a key at logstash-forwarder.key and the 1-year valid certificate at logstash-forwarder.crt. Both the server that is running logstash-forwarder as well as the logstash instances receiving logs will require these files on disk to verify the authenticity of messages. That means we have to distribute it also on the Tandem (the sending party). The logstash forwarder also needs a Java Keystore. We create a new one with the self-signed certificate

    keytool -importcert -trustcacerts -file logstash-forwarder.crt -alias ca -keystore keystore.jks
    

    The command will ask for a password, just the use the default changeit for simplicity. You may choose another password, but keep in mind to remember it.

    Configure logstash

    Logstash, that runs on the Linux Server, needs a lumberjack input configuration:

    input {
      lumberjack {
        port => 5400
        ssl_certificate => "/opt/logstash-2.2.1/logstash-forwarder.crt"
        ssl_key => "/opt/logstash-2.2.1/logstash-forwarder.key"
      }
    }
    

    We just choose the free port 5400 for simplicity. The output may be elasticsearch or for testing just stdout.

    output {
        elasticsearch {
            host => "10.24.62.120"
            protocol => "http"
            port => 9200
            index => "tandem-%{+YYYY.MM.dd}"
        }
        stdout {
            codec => rubydebug
        }
    }
    

    Of course can also apply custom filters, but for simplicity I leave it out the equation.

    The HP NonStop side

    The first obstacle under OSS is to setup the correct Java environment:

    export JAVA_HOME=/usr/tandem/java7.0
    export PATH=$PATH:$JAVA_HOME/bin
    

    Allowing programmes to use the TCP/IP stack is a special case, and had to be done:

    add_define =tcpip^process^name class=map file=\$ZKIP
    

    We assign the current OSS to the process name $ZKIP, that allows us to talk with the Linux Server on the outgoing site. You may have to replace the process name with your respective process name on your Tandem/HP NonStop. Download the latest release from above github repository and upload it to the HP NonStop.

    Configure the forwarder

    I put the SSL certificates under the same folder of the logstash-forwarder. The forwarder needs a configuration, which files he should tail and forward to. An example:

    {
       "network": {
         "servers": [ "10.24.62.120:5400" ],
         "ssl certificate": "/opt/logstash-forwarder/logstash-forwarder.crt",
         "ssl key": "/opt/logstash-forwarder/logstash-forwarder.key",
         "ssl ca": "/opt/logstash-forwarder/keystore.jks",
         "timeout": 15
       },
       "files": [
         {
           "paths": [
             "/var/dev/log/tomcat-server/-*.log"
           ],
           "fields": { "type": "logs" }
         }, {
           "paths": [
             "/var/dev/log/java/*.log"
           ],
           "fields": { "type": "logs" }
         }
       ]
     }
    

    Start the forwarder

    After that we can start the java logstash forwarder with the defined configuration:

    nohup java -jar logstash-forwarder-java-0.2.3.jar -config config > forwarder.log 2> error.log &
    

    On the receiving site or Kibana you should see the incoming messages flying in.

    Final steps

    After testing successfully the log forwarding you may configure a new pathway server to run the application automatically.

  11. 2016-03-19 - Delete Elasticsearch index-name with special characters; Tags: Delete Elasticsearch index-name with special characters
    Loading...

    Delete Elasticsearch index-name with special characters

    If you working with logstash and set the index name dynamically from variables, you might run into situations where the variable is not substituted due to errors. You can get rid of this bad indices by deleting them with the REST API of elasticsearch. The challenge lies in escaping the special characters.

    An example what might happen:

    curl -s http://localhost:9200/_cat/indices
    green open  .marvel-es-2016.03.16        1 1  750253 5159 717.1mb 358.5mb
    green open  fo-%{environment}-2016.03.19 5 1     186    0   346kb   173kb
    green open  .marvel-es-2016.03.17        1 1  772223 5592 726.5mb   367mb
    green open  .marvel-es-2016.03.18        1 1  753400 3932   777mb 388.4mb
    

    To delete the second index we have to escape

    • % → %25
    • { → %7B
    • } → %7D
    curl -XDELETE http://localhost:9200/fo-%25%7Benvironment%7D-2016.03.19
    
  12. 2016-02-15 - Major logstash changes in the email output plugin; Tags: Major logstash changes in the email output plugin
    Loading...

    Major logstash changes in the email output plugin

    Today I was looking on the upgrade of logstash 1.5.4 to newest version 2.2.0 in the office. There are some configuration breakers. One of them, is in the email output plugin.

    The current configuration in logstash 1.5.x allows to use a options hash, with the value of setting the smtp host (smtp-ip-or-host):

    email {
      to => "user@gmail.com"
      codec => "plain"
      contenttype => "text/html; charset=UTF-8"
      options => {
        smtpIporHost => "smtp.gmail.com"
        port => 587
        userName => "user@gmail.com"
        password => "secret"
        starttls => true
      }
    }
    

    Base on this github issue, I was able to trace down the source code in Ruby. In the recent documentation (v2.2) is was hard to find, where smtpIporHost has moved. This is the new configuration in v2.2:

    email {
      to => "user@gmail.com"
      codec => "plain"
      contenttype => "text/html; charset=UTF-8"
      address => "smtp.gmail.com"
      port => 587
      userName => "user@gmail.com"
      password => "secret"
      use_tls => true
    }
    

    Except for address and use_tls the attributes have become its own attributes and moved directly into the email configuration.

  13. 2015-12-09 - Logstash handling of multi-line and exceptions in Java log files; Tags: Logstash handling of multi-line and exceptions in Java log files
    Loading...

    Logstash handling of multi-line and exceptions in Java log files

    JBoss Logs contains messages that are spread across over multiple lines. For instance startup messages and exceptions. This post demonstrates how to handle them in logstash and keep them as message part together.

    Use the codec multiline in the input section to pack those messages together.

    input {
         file {
             type => "jboss"
             path => "/var/log/jboss/log/server.log"
             start_position => beginning
             codec => multiline {
                   pattern => "^%{TIMESTAMP_ISO8601}"
                   negate => true
                   what => "previous"
             }
         }
    }
    

    Every line that doesn’t start with a timestamp, will be assigned to the previous line until a new line is detected.

  14. 2015-10-16 - Multiple configuration files for logstash; Tags: Multiple configuration files for logstash
    Loading...

    Multiple configuration files for logstash

    Structure of a config file: A Logstash config file has a separate section for each type of plugin you want to add to the event processing pipeline.

    # basic scheme
    input {}
    filter {}
    output {}
    

    Use input directory: Instead of one config file, you can split each section(s) into separate files. Instead of the file you omit an input directory. Start logstash with input directory

    bin/logstash -f /etc/logstash.d
    
    • you can use a directory (/etc/logstash.d) as the value for the config input
    • will use all files in this directory
    • loaded in lexicographical order of the files

    Example layout: Contents of /etc/logstash.d

    1_file-input.conf
    1_udp-input.conf
    2_filter.conf
    3_debug-output.conf
    4_elasticsearch-output.conf
    
  15. 2015-10-08 - Convert from CSV to JSON with logstash; Tags: Convert from CSV to JSON with logstash
    Loading...

    Convert from CSV to JSON with logstash

    logstash allows with various plugins a quick solution, to convert CSV input to JSON output.

    This test configuration demonstrates a simple example with nested properties.

    # convert from csv to json
    input {
        generator {
            type => 'csv'
            message => '2015-10-08,CH,4711'
            count => 1
        }
    }
    filter {
        csv {
            columns => ['date', 'countryCode', 'value']
        }
        mutate {
            rename => {
                "date" => "[payment][date]"
                "countryCode" => "[payment][origin]"
                "value" => "[payment][value]"
            }
        }
    }
    output {
        stdout {
            codec => rubydebug{}
        }
    }
    output {
        stdout {
            codec => json{}
        }
    }
    
    • With the generator input plugin, we generate a CSV line.
    • The csv filter plugin, allows the parsing of the CSV data.
    • The mutate filter plugin, allows with rename to nest the properties under payment.
    • The output contains two definitions, the rubydebug and the JSON output (pretty printed by me).

    The output

    Logstash startup completed
    {
           "message" => [
            [0] "2015-10-08,CH,4711"
        ],
          "@version" => "1",
        "@timestamp" => "2015-10-08T14:31:03.344Z",
              "type" => "csv",
              "host" => "cinhtau",
          "sequence" => 0,
           "payment" => {
              "date" => "2015-10-08",
            "origin" => "CH",
             "value" => "4711"
        }
    }
    {
        "message":["2015-10-08,CH,4711"],
        "@version":"1",
        "@timestamp":"2015-10-08T14:31:03.344Z",
        "type":"csv",
        "host":"cinhtau",
        "sequence":0,
        "payment":{
            "date":"2015-10-08",
            "origin":"CH",
            "value":"4711"
        }
    }
    Logstash shutdown completed
    
  16. 2015-09-29 - List open files of a dedicated process; Tags: List open files of a dedicated process
    Loading...

    List open files of a dedicated process

    An open file may be a regular file, a directory, a block special file, a character special file, an executing text reference, a library, a stream or a network file (Internet socket, NFS file or UNIX domain socket.) A specific file or all the files in a file system may be selected by path.

    Linux man page

    The command lsof allows you to display all open files used by a process. Following example uses a logstash process. It has no input files, but usually if you want to know which log files logstash or the logstash forwarder is using, you can display them with lsof. Check which pid logstash has

    tan@cinhtau:/opt/logstash$ ps -Af | grep [l]ogstash
    logstash  2551     1 86 14:18 pts/1    00:00:21 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/var/lib/logstash -Xmx500m -Xss2048k -Djffi.boot.library.path=/opt/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/var/lib/logstash -Xbootclasspath/a:/opt/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/opt/logstash/vendor/jruby -Djruby.lib=/opt/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /opt/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log
    

    Display open files

    tan@cinhtau:/opt/logstash$ sudo lsof -nPp 2551
    COMMAND  PID     USER   FD   TYPE             DEVICE SIZE/OFF    NODE NAME
    java    2551 logstash  cwd    DIR                8,1     4096 1704261 /var/lib/logstash
    java    2551 logstash  rtd    DIR                8,1     4096       2 /
    java    2551 logstash  txt    REG                8,1     7526  931275 /usr/lib/jvm/zulu-8-amd64/jre/bin/java
    java    2551 logstash  mem    REG                8,1    48909  931327 /usr/lib/jvm/zulu-8-amd64/jre/lib/amd64/libmanagement.so
    java    2551 logstash  mem    REG                8,1     1281 1191239 /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.6.5/lib/JRubyFileExtension.jar
    # output truncated a bunch of java libs
    java    2551 logstash   46u  0000                0,9        0    8530 anon_inode
    java    2551 logstash   47r  FIFO                0,8      0t0   23158 pipe
    # output truncated a bunch of pipes
    java    2551 logstash   98u  IPv6              23166      0t0     TCP *:9300 (LISTEN)
    java    2551 logstash   99u  unix 0xffff8801f1b20000      0t0   23167 socket
    java    2551 logstash  100u  IPv6              25071      0t0     TCP 10.0.2.15:54667->10.0.2.15:9300 (ESTABLISHED)
    # output truncated a bunch of used sockets
    java    2551 logstash  129u  IPv6              23259      0t0     TCP 127.0.0.1:9300->127.0.0.1:41905 (ESTABLISHED)
    
  17. 2015-09-29 - Generate random log events for logstash; Tags: Generate random log events for logstash
    Loading...

    Generate random log events for logstash

    Instead of using stdin, logstash also offers the generator input plugin for generating random log events. You can either use a single message or multiple messages in lines.

    This logstash configuration generates emits a single time the defined message. You can check the performance of the filter plugin setup.

    input {
        generator {
            type => 'metrics'
            message => '2015-09-29 09:36:28,574 metrics-memory-stat 2036334592;2147483648;2036334592;439658224'
            count => 1
        }
    }
    filter {
      if [type] == 'metrics' {
        grok {
            match => { 'message' => '(?<datetime>(%{TIMESTAMP_ISO8601})) (?<stat>(.*)) (?<data>(.*))' }
        }
        if [stat] == "metrics-memory-stat" {
            csv {
                source => "data"
                columns => ['heap-committed', 'heap-init', 'heap-max', 'heap-used']
                separator => ";"
            }
            mutate {
                convert => {
                    "heap-committed" => "integer"
                    "heap-init" => "integer"
                    "heap-max" => "integer"
                    "heap-used" => "integer"
                }
            }
        }
        date {
            match => [ 'datetime', 'yyyy-MM-dd HH:mm:ss,SSS', 'HH:mm:ss,SSS', 'ISO8601' ]
            remove_field => [ 'message', 'datetime', 'ISO8601_TIMEZONE', 'data' ]
        }
      }
    }
    output {
        stdout {
            codec => rubydebug{}
        }
    }
    

    This will result in

    Logstash startup completed
    {
              "@version" => "1",
            "@timestamp" => "2015-09-29T07:36:28.574Z",
                  "type" => "metrics",
                  "host" => "cinhtau",
              "sequence" => 0,
                  "stat" => "metrics-memory-stat",
        "heap-committed" => 2036334592,
             "heap-init" => 2147483648,
              "heap-max" => 2036334592,
             "heap-used" => 439658224
    }
    Logstash shutdown completed
    
  18. 2015-09-29 - Grok debugger; Tags: Grok debugger
    Loading...
  19. 2015-09-28 - logstash configuration for Dropwizard Metrics; Tags: logstash configuration for Dropwizard Metrics
    Loading...

    logstash configuration for Dropwizard Metrics

    Dropwizard Metrics generates application metrics for Java applications. This post demonstrates an example of the Slf4jReporter and its logstash configuration.

    grokking the output

    Depending on your log config the Slf4jReporter creates following output:

    16:16:10,966 [metrics] type=TIMER, name=delivery.header.parsing.time, count=1, min=9.517406, max=9.517406, mean=9.517406, stddev=0.0, median=9.517406, p75=9.517406, p95=9.517406, p98=9.517406, p99=9.517406, p999=9.517406, mean_rate=0.03727438239447458, m1=0.1433062621147579, m5=0.18710139700632353, m15=0.19560457449692012, rate_unit=events/second, duration_unit=milliseconds
    

    Using the grok debugger, gives you a grok pattern, which will parsed the log data.

    %{TIME} \[(?<logthread>(?:[a-z]*))\] %{GREEDYDATA}
    

    The GREEDYDATA pattern might not be known on logstash, so it we changed it into:

    %{TIME} \[(?<logthread>(?:[a-z]*))\] (?<data>(.*))
    

    This results in

    {
      "TIME": [
        [
          "16:16:10,966"
        ]
      ],
      "HOUR": [
        [
          "16"
        ]
      ],
      "MINUTE": [
        [
          "16"
        ]
      ],
      "SECOND": [
        [
          "10,966"
        ]
      ],
      "logthread": [
        [
          "metrics"
        ]
      ],
      "data": [
        [
          "type=TIMER, name=delivery.header.parsing.time, count=1, min=9.517406, max=9.517406, mean=9.517406, stddev=0.0, median=9.517406, p75=9.517406, p95=9.517406, p98=9.517406, p99=9.517406, p999=9.517406, mean_rate=0.03727438239447458, m1=0.1433062621147579, m5=0.18710139700632353, m15=0.19560457449692012, rate_unit=events/second, duration_unit=milliseconds"
        ]
      ]
    }
    

    logstash test configuration

    With the know grok pattern, we take the data field as input for the key value filter plugin, that parses the data. We need to configure it properly with a custom field split.

    input {
        stdin {}
    }
    filter {
        grok {
            match => { "message" => "%{TIME} \[(?<logthread>(?:[a-z]*))\] (?<data>(.*))" }
        }
        kv {
            source => "data"
            field_split => ", "
        }
    }
    output {
        stdout {
            codec => "rubydebug"
        }
    }
    

    It creates for each key the respective field:

    {
              "message" => "16:16:10,966 [metrics] type=TIMER, name=delivery.header.parsing.time, count=1, min=9.517406, max=9.517406, mean=9.517406, stddev=0.0, median=9.517406, p75=9.517406, p95=9.517406, p98=9.517406, p99=9.517406, p999=9.517406, mean_rate=0.03727438239447458, m1=0.1433062621147579, m5=0.18710139700632353, m15=0.19560457449692012, rate_unit=events/second, duration_unit=milliseconds\r",
             "@version" => "1",
           "@timestamp" => "2015-09-28T10:28:51.620Z",
                 "host" => "cinhtau",
            "logthread" => "metrics",
                 "data" => "type=TIMER, name=delivery.header.parsing.time, count=1,min=9.517406, max=9.517406, mean=9.517406, stddev=0.0, median=9.517406, p75=9.517406, p95=9.517406, p98=9.517406, p99=9.517406, p999=9.517406, mean_rate=0.03727438239447458, m1=0.1433062621147579, m5=0.18710139700632353, m15=0.19560457449692012, rate_unit=events/second, duration_unit=milliseconds\r",
                 "type" => "TIMER",
                 "name" => "delivery.header.parsing.time",
                "count" => "1",
                  "min" => "9.517406",
                  "max" => "9.517406",
                 "mean" => "9.517406",
               "stddev" => "0.0",
               "median" => "9.517406",
                  "p75" => "9.517406",
                  "p95" => "9.517406",
                  "p98" => "9.517406",
                  "p99" => "9.517406",
                 "p999" => "9.517406",
            "mean_rate" => "0.03727438239447458",
                   "m1" => "0.1433062621147579",
                   "m5" => "0.18710139700632353",
                  "m15" => "0.19560457449692012",
            "rate_unit" => "events/second",
        "duration_unit" => "milliseconds\r"
    }
    

    Pay attention when dumping to elasticsearch the data type is String and needs to be probably converted with the mutate filter plugin or the respective mapping on elasticsearch needs to be modified.

  20. 2015-08-25 - Perform Health-Check on Database Connections with logstash; Tags: Perform Health-Check on Database Connections with logstash
    Loading...

    Perform Health-Check on Database Connections with logstash

    This section describes how to perform health checks on database instances. logstash offers the jdbc input plugin.

    Type Url
    Source Code github
    Reference elastic reference

    Plugin Installation

    At the current logstash version (1.5.3) the jdbc plugin is not included in the default distribution. To install the plugin invoke the ruby gem installation. The plugin name is logstash-input-jdbc. Following example was done on a windows machine.

    bin\plugin.bat install logstash-input-jdbc
    

    Configuration

    This section shows how to perform a health check on the database instance (Oracle DB 11g).

     input {
      jdbc {
        jdbc_driver_library => "C:/Daten/tools/logstash-1.5.3/db/ojdbc6.jar"
        jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
        jdbc_connection_string => "jdbc:oracle:thin:@//localhost:1521/db"
        jdbc_user => "scott"
        jdbc_password => "tiger"
        # will execute on the 0th minute of every hour every day
        # uncomment for testing purpose
        # schedule => "0 * * * *"
        statement => "select 1 from dual"
        jdbc_paging_enabled => "true"
        jdbc_page_size => "1"
        jdbc_validate_connection => "true"
     }
    }
    output {
        stdout { codec => rubydebug }
    }
    

    Pay attention to add the prefix Java:: to the driver class! See also github issue #43

    Testing

    This will give you

    C:\Daten\tools\logstash-1.5.3>bin\logstash.bat -f jdbc.conf
    io/console not supported; tty will not be manipulated
    Logstash startup completed
    {
                 "1" => 1.0,
          "@version" => "1",
        "@timestamp" => "2015-08-18T11:24:37.865Z"
    }
    Logstash shutdown completed
    
  21. 2015-08-24 - Convert multiple fields with mutate plugin for logstash; Tags: Convert multiple fields with mutate plugin for logstash
    Loading...

    Convert multiple fields with mutate plugin for logstash

    The mutate plugin allows to convert fields into other data types. The existing example does not illustrate how to convert multiple fields.

    Find below an example, how to convert multiple fields from string into number formats.

    filter {
        csv {
            source => "data"
            columns => ['OpenFileDescriptorCount', 'MaxFileDescriptorCount', 'CommittedVirtualMemorySize', 'TotalSwapSpaceSize', 'FreeSwapSpaceSize', 'ProcessCpuTime', 'FreePhysicalMemorySize', 'TotalPhysicalMemorySize', 'SystemCpuLoad', 'ProcessCpuLoad', 'AvailableProcessors', 'Arch', 'SystemLoadAverage', 'Version', 'Name', 'ObjectName']
            separator => ";"
        }
        mutate {
            convert => {
                "OpenFileDescriptorCount" => "integer"
                "MaxFileDescriptorCount" => "integer"
                "CommittedVirtualMemorySize" => "integer"
                "TotalSwapSpaceSize" => "integer"
                "FreeSwapSpaceSize" => "integer"
                "ProcessCpuTime" => "integer"
                "FreePhysicalMemorySize" => "integer"
                "TotalPhysicalMemorySize" => "integer"
                "SystemCpuLoad" => "float"
                "ProcessCpuLoad" => "float"
                "AvailableProcessors" => "integer"
                "SystemLoadAverage" => "float"
            }
        }
    }
    

    See also Discuss Elastic

  22. 2015-06-23 - Custom grok patterns for logstash; Tags: Custom grok patterns for logstash
    Loading...

    Custom grok patterns for logstash

    grok is a powerful filter plugin. It allows you to use regular expressions (regex). This post gives some advices how to test and develop custom grok patterns for logstash.

    First, you can use the Oniguruma syntax for named capture which will let you match a piece of text and save it as a field:

    (?<field_name>the pattern here)
    

    Oniguruma is the regular expression engine that is built-in in Ruby since v1.9.

    Having a log entry

    09:33:45,416 (metrics-logger-reporter-1-thread-1) type=GAUGE, name=notifications.received, value=2
    

    You can use the grok pattern

    (?<logtime>%{HOUR}:%{MINUTE}:%{SECOND}) (?<logthread>(?:[()a-zA-Zd-]+)) type=(?<type>([A-Z])w+), name=(?<name>(?:[A-Za-z.]*)w+), value=(?<value>([0-9]|NaN)+)
    

    Pay attention that the value might be number or NaN. It is easier to use the kv (key-value) plugin, but this is solely for demonstration. You may also define custom patterns from the retrieved regular expressions, but this won’t be handled in this article.

    The logstash configuration

    filter {
      grok {
        match => { "message" => "(?<logtime>%{HOUR}:%{MINUTE}:%{SECOND}) (?<logthread>(?:[()a-zA-Zd-]+)) type=(?<type>([A-Z])w+), name=(?<name>(?:[A-Za-z.]*)w+), value=(?<value>([0-9]|NaN)+)" }
      }
    }
    

    Regular expressions itself may be a complicated issue, therefore there is a fantastic online ruby regular expression editor: Rubular to assist you finding the right regex.

    Rubular Editor

    You can also use the grok debugger for testing the regex.

    Grok Debugger

  23. 2015-06-22 - Filter with grok; Tags: Filter with grok
    Loading...

    Filter with grok

    grok is a filter plugin and “is perfect for syslog logs, apache and other webserver logs, mysql logs, and in general, any log format that is generally written for humans and not computer consumption.”

    grok = understand (something) intuitively or by empathy. Source: The Oxford English Dictionary

    It has a preset of existing patterns. Assumed you have in your Apache HTTP Server configuration:

    CustomLog ${APACHE_LOG_DIR}/access.log combined
    

    A log entry example

    81.62.38.214 - - [17/Jun/2015:09:27:37 +0200] "GET /wp/ HTTP/1.1" 200 7141 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0"
    

    We use for the logstash configuration the grok pattern COMBINEDAPACHELOG which consists of:

    # Log formats
    COMMONAPACHELOG %{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)
    COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}
    

    To test the matching, we configure in the logstash configuration the output as json and used the parsed value clientip as input for another filter plugin geoip, to determine the origin country of given IP address.

    input {
      stdin {}
    }
    filter {
      grok {
        match => { "message" => "%{COMBINEDAPACHELOG}" }
      }
      geoip {
        source => "clientip"
      }
    }
    output {
      stdout { codec => json }
    }
    

    This will result in:

    cinhtau@edge:~/test/logstash/bin$ ./logstash -f apache.conf
    Logstash startup completed
    81.62.38.214 - - [17/Jun/2015:09:27:37 +0200] "GET /wp/ HTTP/1.1" 200 7141 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0"
    {"message":"81.62.38.214 - - [17/Jun/2015:09:27:37 +0200] "GET /wp/ HTTP/1.1" 200 7141 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0"","@version":"1","@timestamp":"2015-06-22T12:15:42.373Z","host":"edge","clientip":"81.62.38.214","ident":"-","auth":"-","timestamp":"17/Jun/2015:09:27:37 +0200","verb":"GET","request":"/wp/","httpversion":"1.1","response":"200","bytes":"7141","referrer":""-"","agent":""Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0"","geoip":{"ip":"81.62.38.214","country_code2":"CH","country_code3":"CHE","country_name":"Switzerland","continent_code":"EU","latitude":47.0,"longitude":8.0,"timezone":"Europe/Zurich","location":[8.0,47.0]}}
    

    If you have to specify custom patterns, that not exists in predefined sets (which is sad but may occur), use the online Grok Debugger to test the matching.

  24. 2015-06-19 - Test your logstash configuration; Tags: Test your logstash configuration
    Loading...

    Test your logstash configuration

    The logstash configuration can become quite complex. Some hints how to tryout if the configuration is not working.

    • use the --configtest option
    • use the --debug option
    • use the stdout output plugin to check the outcome of the filter plugins

    Use --configtest

    C:\tools\logstash-1.5.1\bin>logstash.bat -f demo.conf --configtest
    Configuration OK
    

    logstash under Linux with --debug option

    ./logstash -f demo.conf --debug
    

    Check the outcome of input and filter using stdout, additionally with output as json

    output {
    	# stdout {}
    	stdout { codec => "json" }
    }
    

    Another codec of interest:

    stdout { codec => "rubydebug" }
    
  25. 2015-06-19 - Send JSON from logstash to Elasticsearch; Tags: Send JSON from logstash to Elasticsearch
    Loading...

    Send JSON from logstash to Elasticsearch

    To send data from a RESTful Webservice, there is the input plugin exec. It executes a command for a defined interval.

    Using this logstash.conf produced a json parse failure in Elasticsearch.

    input {
    	exec {
    		command => "C:\Daten\tools\logstash-1.5.1\bin\metrics.bat"
    		interval => 10
    		codec => json
    	}
    }
    output {
    	elasticsearch {
    	   node_name => test
    	   host => localhost
    	}
    }

    Elasticsearch receives this message:

    C:\bin\curl\curl.exe http://localhost:8080/jolokia/read/metrics:name=trx.process.approved
    {"request":"mbean":"metrics:name=trx.process.approved","type":"read"},"value":{"Count":14},"timestamp":1434641808,"status":200}

    Well that’s not json! As the docs says:

    The @message of this event will be the entire stdout of the command as one event.

    My solution is to use the filter plugins split(), drop() and json().

    input {
       exec {
    		command => "C:\Daten\tools\logstash-1.5.1\bin\metrics.bat"
    		interval => 10
    		codec => plain
       }
    }
    filter {
       split {
       }
       if  [message] =~ "^{" {
    		# convert to json
    		json {
    			 source => "message"
    		}
       }
       else {
    		# drop all lines that are not json
    		drop {}
       }
    }
    output {
    	elasticsearch {
    	   node_name => test
    	   host => localhost
    	}
    }
    • split will take the message and parse it into three separate files
    • check with a regex if the line begins with the json delimiter “{“
    • if the line is the json part convert it to json and let elasticsearch parse it properly
    • else drop the line
  26. 2015-06-19 - logstash / or ?; Tags: logstash / or ?
    Loading...

    logstash / or ?

    Working with logstash on a Windows machine is a little tricky. The curl command exists as Windows binary. Invoking curl gives me a JSON message, that I used for logstash input.

    curl.exe http://localhost:8080/jolokia/read/metrics:name=trx.process.approved

    Output

    {
        "request": {
            "mbean": "metrics:name=trx.process.approved",
            "type": "read"
        },
        "value": {
            "Count": 455
        },
        "timestamp": 1434716739,
        "status": 200
    }

    Configure the logstash.conf

    input {
        exec {
            command => "C:\bin\curl\curl.exe http://localhost:8080/jolokia/read/metrics:name=trx.process.approved"
            interval => 10
            codec => json
        }
    }
    output {
        stdout {}
    }

    Should work, right? Nope. Running logstash:

    logstash.bat -f logstash.conf
    Logstash startup completed
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
      0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
    curl: (6) Could not resolve host: http://localhost:8080/jolokia/read/metrics

    Seems that on Windows / was replaced with `` and the string after “:” missed. Escaping didn’t work. The final solution is to write a .bat file that invokes that command, to avoid the issue.

    New logstash.conf using metrics.bat that contains the curl command.

    input {
        exec {
            command => "C:\tools\logstash-1.5.1\bin\metrics.bat"
            interval => 10
            codec => json
        }
    }
    output {
        stdout {}
    }

    Basic conclusion: Avoid using Logstash with Windows. If you intend to use Logstash as log shipper on a Linux Server, test it there.

    Update 2018: In 2015 the Beats family wasn’t around.