elasticsearch - Logstash filter - half json line parse -

i'm using 'filebeat' shipper client send redis, read redis logstash , send es.

i'm trying parse following example line:

09:24:01.969 watchdog - info - 100.140.2 passed: mobile:mobile[].popover["mc1814"].select(2,) :706<<<<<<<<<<<<<<<<<<< {"actionduration":613}

in end want have field names: "actionduration" value: 613.

as can see it's partially json. - i've tried use grok filter, add_field , match , i've tried change few configurations in filebeat , logstash.

i'm using basic configurations: filebeat.conf:

filebeat.prospectors:

• input_type: log

  paths:

  • /sketch/workspace/sanity-dev-kennel/out/*.log

  fields:

  • type: watchdog
  • build_id: 82161

if there's possibility in filebeat side prefer, it's in logstash side.

thanks lot, moshe

this sort of partial-formatting best handled on logstash side, not shipper. filters/transforms available in filebeat aren't that. logstash filter pipeline is, though.

filter {   grok {     match => {       "message" => [ "(?<plain_prefix>^.*?) (?<json_segment>{.*$)"]     }   }   json {     source => "json_segment"   }   mutate {     remove_field => [ "json_segment" ]   } } 

this basic example split incoming message 2 fields. plain_prefix , json_segment. json{} filter used parse json data event. finally, mutate {} filter used remove json_segment field event, has been parsed , included.

note: .*? in plain_prefix critical in filter. constructed way, first { onward considered part of json segment. if use .*, json segment last {, problem complex json datastructures.