[FLUME-2779] MultiLine Deserializer for Spooling DIrectory Source

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: notrack
• Component/s: Sinks+Sources
• Flags: Patch
• Assignee: Unassigned
• Reporter: Siva Kotipalli
• Created: 30/Aug/15 15:56
• Updated: 19/Sep/15 17:28

Description

1. Spool Directory source defaults to LINE, this has issue when multiline log events like stack traces and have request/responses.
2. Proposal to have log line start regex Key to aggregate all the log lines till the next regex key is found.
3. As there is aggregation needed based on startKey, maxLineLength is dropped on deserializer and included maxLogLineLength as alternative, this basically truncates the event based on the length and remaining part is never transferred to sink.
4. logstartregexkey is introduced on Deserializer to include the start regex key.
5. Patch available with unit tests.