[METRON-1761] Allow a grok statement to be applied to each line in a file.

• Type: Improvement
• Status: Done
• Priority: Minor
• Resolution: Done
• Fix Version/s: 0.7.0
• Assignee: Otto Fowler
• Reporter: Laurens Vets
• Created: 31/Aug/18 17:27
• Updated: 11/Dec/18 16:09
• Resolved: 10/Oct/18 16:20

Description

Make grok work where each line in incoming logs is a separate unit to be parsed.

This would for instance allow NiFi to pick up log files (whereby each line is to be parsed separately) and send them to Metron without having to split the content.

Example content of a log file where a grok statement needs to be applied to each line:

2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 0.000073 0.001048 0.000057 200 200 0 29 "GET http://www.example.com:80/ HTTP/1.1" "curl/7.38.0" - -
2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 0.000086 0.001048 0.001337 200 200 0 57 "GET https://www.example.com:443/ HTTP/1.1" "curl/7.38.0" DHE-RSA-AES128-SHA TLSv1.2
2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 0.001069 0.000028 0.000041 - - 82 305 "- - - " "-" - -
2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 0.001065 0.000015 0.000023 - - 57 502 "- - - " "-" ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2

Issue Links

• links to: GitHub Pull Request #1184