[FLUME-2182] Spooling Directory Source will not ingest data completely when a wide character appears at the edge of a buffer

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 1.5.0
• Component/s: Sinks+Sources
• Assignee: Sven Meys
• Reporter: syntony liu
• Created: 05/Sep/13 14:01
• Updated: 29/Sep/14 22:37
• Resolved: 28/Sep/13 00:19

Description

the bug is in ResettableFileInputStream.java: int readChar().
if the last byte of buf is only a partial of a wide character, readChar() shouldn't return -1(ResettableFileInputStream.java:186). it
loses the remanent data in a file.

I fix it such as:
public synchronized int readChar() throws IOException {
// if (!buf.hasRemaining()) {
if(buf.limit()- buf.position < 10)

{ refillBuf(); }

int start = buf.position();
charBuf.clear();

boolean isEndOfInput = false;
if (position >= fileSize)

{ isEndOfInput = true; }

CoderResult res = decoder.decode(buf, charBuf, isEndOfInput);
if (res.isMalformed() || res.isUnmappable())

{ res.throwException(); }

int delta = buf.position() - start;

charBuf.flip();
if (charBuf.hasRemaining())

{ char c = charBuf.get(); // don't increment the persisted location if we are in between a // surrogate pair, otherwise we may never recover if we seek() to this // location! incrPosition(delta, !Character.isHighSurrogate(c)); return c; // there may be a partial character in the decoder buffer }

else

{ incrPosition(delta, false); return -1; }

}

it avoid a partial character, but have new issue. sometime, some lines of a log file have a repeated character.
eg.
original file: 123456
sink file: 1233456

Issue Links

• duplicates: FLUME-2215 ResettableFileInputStream can't support ucs-4 character
• is duplicated by: FLUME-2241 Spooling Directory Source doesn't handle 2 byte UTF-8 encoded characters correctly
• relates to: FLUME-2241 Spooling Directory Source doesn't handle 2 byte UTF-8 encoded characters correctly