[FLUME-2215] ResettableFileInputStream can't support ucs-4 character

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.5.0
• Fix Version/s: 1.7.0
• Labels: patch
• Assignee: Alexandre Dutra
• Reporter: syntony liu
• Created: 16/Oct/13 08:10
• Updated: 04/Jan/17 09:01
• Resolved: 27/May/15 17:04

Description

ResettableFileInputStream.java:readChar() not handle ucs-4 character. it need 2 charBuf. it cause an unexpected termination。
a temporary solution:
if (res.isOverflow() && !charBuf.hasRemaining()){
logger.warn("decoder ucs-4 at postion: {}" , buf.position());
tmpBuf.clear();
res = decoder.decode(buf, tmpBuf, isEndOfInput);
incrPosition( buf.position() - start, false);
return '?';
}

Issue Links

• is duplicated by: FLUME-2360 Spooling Source: LineDeserializer incorrectly stops reading events if UTF-8 char occurs at end of buffer
• is duplicated by: FLUME-2182 Spooling Directory Source will not ingest data completely when a wide character appears at the edge of a buffer
• is duplicated by: FLUME-2241 Spooling Directory Source doesn't handle 2 byte UTF-8 encoded characters correctly
• relates to: FLUME-2535 DEFAULT_BUF_SIZE hits special unicode character on the last byte truncates the event stream
• relates to: FLUME-654 how to let flume support Chinese log file
• links to: Review board
• links to: Review Request for patch 0
• links to: Review Request for patch 1