Details
-
Type:
Improvement
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 1.9.0
-
Component/s: None
-
Labels:None
Description
Flume HDFS Sink will currently only attempt to close a file once, and if that fails, it will move to recoverLease. This appears to be causing some issues.
The recommendation is to modify the close behavior to retry the close a configurable number of times with a configurable interval in between before attempting the recoverLease.
There currently exists a closeTries and retryInterval property, however this only retries the rename operation, not the close operation as seen in the documentation:
The file may still remain open if the close call fails but the data will be intact and in this case, the file will be closed only after a Flume restart.
