[FLUME-1988] Add Support for Additional Deserializers for SpoolingDirectorySource

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.4.0
• Component/s: Docs, Sinks+Sources
• Labels: serializers

Release Note

org.apache.flume.serialization.RegexDelimiterDeSerializer
org.apache.flume.serialization.TestRegexDelimiterDeSerializer

Splits input into event "blocks" based on a configurable regex pattern.
The pattern is included in the block it is split on, so generally the user will want to split on something that marks the end

• Assignee: Israel Ekpo
• Reporter: Israel Ekpo
• Due: 30/Apr/13
• Created: 11/Apr/13 11:23
• Updated: 09/Apr/21 18:41

Description

There are certain use cases for SpoolingDirectorySource where the events in the log file are not delimited with newline characters.

Certain log files that contain stack traces, xml documents and pretty JSON strings seem to contain multiple new line characters within each event.

We can use alternative logic such as specific characters, strings or regular expressions to determine when the event is complete.

Hence I am proposing the following new deserializers based on org.apache.flume.serialization.LineDeserializer

1. org.apache.flume.serialization.RegexDelimiterDeSerializer
   Allows the user to specify a regular expression that is a delimiter for events within the log file

1. org.apache.flume.serialization.CharSequenceDelimiterDeSerializer
   Allows the user to specify a comma separated character sequence that is a delimiter for events within the log file
   The user will specify an integer for the ascii characters and we will use that as the delimter.
   For example support for \r\n could be specified as 13,10
   A list of codes is available at http://www.asciitable.com/

We will also need to update the user guide with examples on how to configure and specify a custom deserializer.

Sub-Tasks

• Source Code and Unit Tests for org.apache.flume.serialization.RegexDelimiterDeSerializer: Open
• Source Code and Unit Tests for org.apache.flume.serialization.CharSequenceDelimiterDeSerializer: Open
• Documentation for New Deserializers for SpoolingDirectorySource: Open