Loading...

Parse XML content with Logstash

Used:

  • logstash 6.1.2

A customer of mine, requires xml data as separate field data for further investigation. The data itself is part of a log message that is processed by Logstash. Logstash provides the powerful XML filter plugin for further parsing.

A test configuration `xml-filter.conf

input {
  stdin {
    codec => plain
  }
}
filter {
  grok {
    match => {
      "message" => "body=%{GREEDYDATA:xml}"
    }
  }
  xml {
    source => "xml"
    target => "body"
    remove_field => ["message","xml"]
  }
}
output {
  stdout { codec => "rubydebug" }
}

Start logstash with above config:

bin/logstash -f xml-filter.conf

[2018-02-01T11:32:06,963][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>250, :thread=>"#<Thread:0x517d9c29 run>"}
[2018-02-01T11:32:07,045][INFO ][logstash.inputs.stdin    ] Automatically switching from plain to line codec {:plugin=>"stdin"}
[2018-02-01T11:32:07,148][INFO ][logstash.pipeline        ] Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[2018-02-01T11:32:07,328][INFO ][logstash.agent           ] Pipelines running {:count=>1, :pipelines=>["main"]}

Paste this example input into the console:

response: header={ "headerVersion" : "3.0", "specVersion" : "6.3.0", "securityLevel" : "MAC", "mac" : "xxx" }, body=<?xml version="1.0" encoding="UTF-8"?><ep2:message xmlns:ep2="http://www.eftpos2000.ch" specversion="0630"><ep2:authrsp msgnum="1141"><ep2:AcqID>4711</ep2:AcqID></ep2:authrsp></ep2:message>

Logstash process it and outputs it:

{
    "@timestamp" => 2018-02-01T10:32:07.397Z,
          "body" => {
        "specversion" => "0630",
            "authrsp" => [
            [0] {
                 "AcqID" => [
                    [0] "4711"
                ],
                "msgnum" => "1141"
            }
        ],
          "xmlns:ep2" => "http://www.eftpos2000.ch"
    },
          "host" => "omega",
      "@version" => "1"
}