[FLUME-1350] HDFS file handle not closed properly when date bucketing

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.1.0, 1.2.0
• Component/s: Sinks+Sources
• Assignee: Unassigned
• Reporter: Robert Mroczkowski
• Created: 08/Jul/12 11:30
• Updated: 29/Oct/12 23:52

Description

With configuration:

agent.sinks.hdfs-cafe-access.type = hdfs
agent.sinks.hdfs-cafe-access.hdfs.path = hdfs://nga/nga/apache/access/%y-%m-%d/
agent.sinks.hdfs-cafe-access.hdfs.fileType = DataStream
agent.sinks.hdfs-cafe-access.hdfs.filePrefix = cafe_access
agent.sinks.hdfs-cafe-access.hdfs.rollInterval = 21600
agent.sinks.hdfs-cafe-access.hdfs.rollSize = 10485760
agent.sinks.hdfs-cafe-access.hdfs.rollCount = 0
agent.sinks.hdfs-cafe-access.hdfs.txnEventMax = 1000
agent.sinks.hdfs-cafe-access.hdfs.batchSize = 1000
#agent.sinks.hdfs-cafe-access.hdfs.codeC = snappy
agent.sinks.hdfs-cafe-access.hdfs.hdfs.maxOpenFiles = 5000
agent.sinks.hdfs-cafe-access.channel = memo-1

When new directory is created previous file handle remains opened. rollInterval setting is used only with files in current date bucket.

Issue Links

• relates to: FLUME-1660 Close "idle" hdfs handles
• links to: Link to the review page