[FLUME-2331] Large TMP files created and never closed

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.4.0
• Component/s: File Channel
• Assignee: Unassigned
• Reporter: Krishna Kumar
• Created: 24/Feb/14 15:41
• Updated: 27/Feb/14 10:34

Description

We are currently writing files to Hadoop partitioned by year, month, day via Flume. File rollovers are done every 5 minutes. Recently, we noticed that this file rollover stops happening sometime during the day and that further data is written to an open TMP file. Because there are no further file rollovers, this TMP file becomes very large. At the end of the day, the TMP is not closed also and Flume goes to the next day, creating new files.

We use a "." prefix to prevent Hive from complaining of the open TMP file. Because of this issue where the TMP file is never closed, the file remains hidden to Hive even after the day ends.