Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-1864

Allow hdfs idle callback to clean up closed bucket writers

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.0
    • Fix Version/s: 1.4.0
    • Component/s: None
    • Labels:
      None

      Description

      In the original implementation of the idle file closing behaviour the callback was cancelled on close(). This makes sense assuming everything else behaves in a desirable manner.

      On the other hand, rollInterval will close a file and leave the bucketWriter in the writer map. This allows for incrementally named files to be created as the same path is reopened. However in some situations(primarily with time bucketed data), this leaves a lot of abandoned bucket writers that may be closed by rollInterval and thus never removed by idle.

      In FLUME-1850 a couple of approaches were suggested, and I originally intended to fix this by using the callback from rollInterval to remove the writer from the map. However this would break incremental naming.

      Until(if we ever do) we change the rolling logic it leaves the more viable option to be allowing the idle timer to persist after a close. In this way it can be used to release resources that have become unnecessary but already closed by rollInterval, also reducing hard to understand interactions between configuration variables by one.

      1. FLUME-1864
        1 kB
        Juhani Connolly

        Issue Links

          Activity

          Hide
          hshreedharan Hari Shreedharan added a comment -

          Committed to trunk, rev: ce05be585f1cb91a3a00ccdfdeaaf5800ada6c35

          Show
          hshreedharan Hari Shreedharan added a comment - Committed to trunk, rev: ce05be585f1cb91a3a00ccdfdeaaf5800ada6c35
          Hide
          hudson Hudson added a comment -

          Integrated in flume-trunk #362 (See https://builds.apache.org/job/flume-trunk/362/)
          FLUME-1864. Allow hdfs idle callback to clean up closed bucket writers. (Revision ce05be585f1cb91a3a00ccdfdeaaf5800ada6c35)

          Result = SUCCESS
          hshreedharan : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=ce05be585f1cb91a3a00ccdfdeaaf5800ada6c35
          Files :

          • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
          Show
          hudson Hudson added a comment - Integrated in flume-trunk #362 (See https://builds.apache.org/job/flume-trunk/362/ ) FLUME-1864 . Allow hdfs idle callback to clean up closed bucket writers. (Revision ce05be585f1cb91a3a00ccdfdeaaf5800ada6c35) Result = SUCCESS hshreedharan : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=ce05be585f1cb91a3a00ccdfdeaaf5800ada6c35 Files : flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java

            People

            • Assignee:
              juhanic Juhani Connolly
              Reporter:
              juhanic Juhani Connolly
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development