[FLUME-2800] Multiline log events for Taildir Source

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 1.6.0, 1.7.0
• Component/s: Sinks+Sources
• Assignee: Unassigned
• Reporter: David Burgos. Isban (Banco Santander)
• Created: 22/Sep/15 09:56
• Updated: 12/Aug/16 13:24
• Resolved: 12/Aug/16 13:23

Description

This a proposal of implementation to handle multiline log messages for new tailing source FLUME-2498.
Based on an idea FLUME-2779 MultiLine Deserializer for Spooling DIrectory Source.

Config.

• multiLineRegex: Regular expression to handle multiline log messages (grok expressions permitted)
• grokDictionaryDir: Custom Grok dictionaries directory
• maxNumberLines: Max number of lines per event in multiline log messages. Default 100. Remaining lines is never transferred to sink.

For Regex expressions use Java regex engine.

Include a functionality for extracting grok expressions into a pure named regex (inspired by the logstash inteceptor)
By default load the included built-in grok dictionaries with pre-defined patterns.
https://github.com/aicer/grok

Attached patch includes a config documentation and unit tests.
Also attached a completed port/patch for Flume 1.6 a Java 1.6

Issue Links

• duplicates: FLUME-2961 Make TaildirSource work with multiline