[FLUME-205] TailSource reads lines using a method(readLine) which does character set interpretation and that breaks all my UTF-8 characters

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.9.1
• Fix Version/s: 0.9.2
• Component/s: Node

Environment

Debian Lenny, with all files in UTF-8 Encoding

• Assignee: Jonathan Hsieh
• Reporter: Disabled imported user
• Created: 06/Sep/10 17:42
• Updated: 13/Dec/10 14:37
• Resolved: 04/Nov/10 21:01

Description

Flume tails a file that is encoded in UTF-8, opening the file shows me ä,ö,ü and others characters. When I open the seq files in Hadoop, which were transmitted and stored by flume through the collectorSink in raw format, all special characters like ä,ö,ü are broken like ä — it seems somewhere might be a change between UTF-8 and another encoding or is the raw output format the problem?

From Jon:
"I think the bug in TailSource – it reads lines using a method (readLine) which does character set interpretation."

Original discussion:
https://groups.google.com/a/cloudera.org/group/flume-user/browse_thread/thread/20231a0f98569d8a#

Issue Links

• is related to: FLUME-252 Update Tail to get rid of races and truncation problems.