[FLUME-2360] Spooling Source: LineDeserializer incorrectly stops reading events if UTF-8 char occurs at end of buffer

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.4.0
• Component/s: Sinks+Sources

Environment

CDH 4.4

• Assignee: Unassigned
• Reporter: Keith Wright
• Created: 10/Apr/14 16:41
• Updated: 04/Nov/14 11:45

Description

We are using a Spooling source to read from files and have noticed that there are situations where the source will stop reading in the middle of a file (see attached example file). After much debugging, I have determine that this occurs if the buffer in ResettableFileInputStream happens to fill when in the middle of handling a UTF-8 character. In this case readChar() will return -1 which ends the read lines loop in LineSerializer. Running the spool source with the attached file should result in > 5400 events in the channel but now it will only return ~3300.

Issue Links

• duplicates: FLUME-2215 ResettableFileInputStream can't support ucs-4 character
• relates to: FLUME-2535 DEFAULT_BUF_SIZE hits special unicode character on the last byte truncates the event stream