[FLUME-734] escapedFormatDfs goes into a file creation frenzy

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.9.4
• Fix Version/s: 0.9.5
• Component/s: Sinks+Sources

Environment

CentOS 5.6

• Assignee: Jonathan Hsieh
• Reporter: Eran Kutner
• Created: 10/Aug/11 09:10
• Updated: 01/Oct/11 00:02
• Resolved: 30/Sep/11 22:57

Description

Using this configuration:
collectorSource(54001) | collector(600000) { escapedFormatDfs("hdfs://hadoop1-m1:8020/raw-events/%Y-%m-%d/", "events-%

{rolltag}

-col1.snappy", seqfile("SnappyCodec")) }

The expected behavior is to see a new file created every 10 minutes. However, once in a while the collector would go into a file creation frenzy, creating new files every second.
The log indicates that writing has failed with error: "OutputFormat instance can only write to the same OutputStream" causing the file to be closed a new one to be opened just to be closed again.

Looking at the code I'm not even sure how the output stream could change but the behavior I'm seeing feels like some sort of a race condition. It is happening much more under heavy load than under low load.

See attached log excerpt.

Issue Links

• is related to: FLUME-692 Too many warnings about deprecated format syntax.