[FLUME-1856] HDFS sink idleTimeout does not close removed BucketWriters

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.4.0, 1.3.1
• Component/s: Sinks+Sources
• Assignee: Gopinathan A
• Reporter: Connor Woodson
• Created: 17/Jan/13 08:28
• Updated: 17/Mar/14 06:09

Description

When a BucketWriter idles out, it is removed from the bucket list but it is never closed. Optimal solution will be to override LinkedHashMap.remove similar to the removeEldestEntry code to close the bucket.

The effects of this not happening is that if you rely on idleTimeout for closing buckets (such as setting no roll period and attempting to roll via timestamped path), as the buckets never get closed the data never goes to s3 and so the heap gets very full.