1. 2015-06-22 - Filter with grok; Tags: Filter with grok

    Filter with grok

    grok is a filter plugin and “is perfect for syslog logs, apache and other webserver logs, mysql logs, and in general, any log format that is generally written for humans and not computer consumption.”

    grok = understand (something) intuitively or by empathy. Source: The Oxford English Dictionary

    It has a preset of existing patterns. Assumed you have in your Apache HTTP Server configuration:

    CustomLog ${APACHE_LOG_DIR}/access.log combined

    A log entry example - - [17/Jun/2015:09:27:37 +0200] "GET /wp/ HTTP/1.1" 200 7141 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0"

    We use for the logstash configuration the grok pattern COMBINEDAPACHELOG which consists of:

    # Log formats
    COMMONAPACHELOG %{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)

    To test the matching, we configure in the logstash configuration the output as json and used the parsed value clientip as input for another filter plugin geoip, to determine the origin country of given IP address.

    input {
      stdin {}
    filter {
      grok {
        match => { "message" => "%{COMBINEDAPACHELOG}" }
      geoip {
        source => "clientip"
    output {
      stdout { codec => json }

    This will result in:

    cinhtau@edge:~/test/logstash/bin$ ./logstash -f apache.conf
    Logstash startup completed - - [17/Jun/2015:09:27:37 +0200] "GET /wp/ HTTP/1.1" 200 7141 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0"
    {"message":" - - [17/Jun/2015:09:27:37 +0200] "GET /wp/ HTTP/1.1" 200 7141 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0"","@version":"1","@timestamp":"2015-06-22T12:15:42.373Z","host":"edge","clientip":"","ident":"-","auth":"-","timestamp":"17/Jun/2015:09:27:37 +0200","verb":"GET","request":"/wp/","httpversion":"1.1","response":"200","bytes":"7141","referrer":""-"","agent":""Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0"","geoip":{"ip":"","country_code2":"CH","country_code3":"CHE","country_name":"Switzerland","continent_code":"EU","latitude":47.0,"longitude":8.0,"timezone":"Europe/Zurich","location":[8.0,47.0]}}

    If you have to specify custom patterns, that not exists in predefined sets (which is sad but may occur), use the online Grok Debugger to test the matching.

  2. 2015-06-22 - IntelliJ Apache configuration support; Tags: IntelliJ Apache configuration support

    IntelliJ Apache configuration support

    If you use IntelliJ or any other IDEA product and have to deal with Apache configurations like .htaccess files, the following plugin might interest you. It supports syntax auto-completing and high-lightning.