Details
-
Improvement
-
Status: Resolved
-
Minor
-
Resolution: Duplicate
-
1.6.0, 1.7.0
-
None
-
None
Description
This a proposal of implementation to handle multiline log messages for new tailing source FLUME-2498.
Based on an idea FLUME-2779 MultiLine Deserializer for Spooling DIrectory Source.
Config.
- multiLineRegex: Regular expression to handle multiline log messages (grok expressions permitted)
- grokDictionaryDir: Custom Grok dictionaries directory
- maxNumberLines: Max number of lines per event in multiline log messages. Default 100. Remaining lines is never transferred to sink.
For Regex expressions use Java regex engine.
Include a functionality for extracting grok expressions into a pure named regex (inspired by the logstash inteceptor)
By default load the included built-in grok dictionaries with pre-defined patterns.
https://github.com/aicer/grok
Attached patch includes a config documentation and unit tests.
Also attached a completed port/patch for Flume 1.6 a Java 1.6
Attachments
Attachments
Issue Links
- duplicates
-
FLUME-2961 Make TaildirSource work with multiline
- Patch Available