Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
1.7.0
-
None
-
None
-
None
Description
Currently the idleFuture which is responsible of closing idle files is scheduled in BucketWriter.flush() while it is cancelled in BucketWriter.append().
If events were appended to a BucketWriter but the next flush failed then the idleFuture won't be scheduled, which lead to unclosed/unrenamed files in a setup where there is no other logic to close/rename the files (i.e. hdfs.rollInterval, hdfs.rollSize and hdfs.rollCount are all 0).
Moreover, if any of the flush() calls fail in HDFSEventSink.process() (https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java#L429) then all the subsequent flushes will be skipped thus no idleFutures will be scheduled.
I'd recommend to move the scheduling of the idleFuture (https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java#L450-L467) to BucketWriter.append().