Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-1864

Allow hdfs idle callback to clean up closed bucket writers

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments


    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.0
    • Fix Version/s: 1.4.0
    • Component/s: None
    • Labels:


      In the original implementation of the idle file closing behaviour the callback was cancelled on close(). This makes sense assuming everything else behaves in a desirable manner.

      On the other hand, rollInterval will close a file and leave the bucketWriter in the writer map. This allows for incrementally named files to be created as the same path is reopened. However in some situations(primarily with time bucketed data), this leaves a lot of abandoned bucket writers that may be closed by rollInterval and thus never removed by idle.

      In FLUME-1850 a couple of approaches were suggested, and I originally intended to fix this by using the callback from rollInterval to remove the writer from the map. However this would break incremental naming.

      Until(if we ever do) we change the rolling logic it leaves the more viable option to be allowing the idle timer to persist after a close. In this way it can be used to release resources that have become unnecessary but already closed by rollInterval, also reducing hard to understand interactions between configuration variables by one.




            • Assignee:
              juhanic Juhani Connolly
              juhanic Juhani Connolly


              • Created:

                Issue deployment