Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
Flume HDFS Sink will currently only attempt to close a file once, and if that fails, it will move to recoverLease. This appears to be causing some issues.
The recommendation is to modify the close behavior to retry the close a configurable number of times with a configurable interval in between before attempting the recoverLease.
There currently exists a closeTries and retryInterval property, however this only retries the rename operation, not the close operation as seen in the documentation:
The file may still remain open if the close call fails but the data will be intact and in this case, the file will be closed only after a Flume restart.
