Flume
  1. Flume
  2. FLUME-2265

Closed bucket writers should be removed from sfwriters map

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v1.5.0
    • Component/s: None
    • Labels:
      None

      Description

      If we don't remove the bucket writer from the map, the reference is kept in the map and not removed till the maxOpenFiles limit is hit, even though the bucket writer is closed. This leads to HDFS buffers sticking around for a long time after the bucket writer is closed.

      1. FLUME-2265.patch
        38 kB
        Hari Shreedharan

        Issue Links

          Activity

          Hide
          Juhani Connolly added a comment -

          Didn't we cover this with the option to close idle files?

          Show
          Juhani Connolly added a comment - Didn't we cover this with the option to close idle files?
          Hide
          Hari Shreedharan added a comment -

          Idle close does remove the reference to the bucket writer from the map. But when a normal file roll occurs - say once every 10 mins or so (and idle is not configured), then the references will remain in the map. If the user does not set idle timeout, then this leads to a leak of the handle until maxOpenFiles is reached. Of course, this is fixed if idleTimeout is set, but we should also handle the case where it is not.

          Show
          Hari Shreedharan added a comment - Idle close does remove the reference to the bucket writer from the map. But when a normal file roll occurs - say once every 10 mins or so (and idle is not configured), then the references will remain in the map. If the user does not set idle timeout, then this leads to a leak of the handle until maxOpenFiles is reached. Of course, this is fixed if idleTimeout is set, but we should also handle the case where it is not.
          Hide
          Hari Shreedharan added a comment -

          Patch that handles this. It changes the way we handle rollInterval-based close. Added a couple of tests to test this as well.

          Show
          Hari Shreedharan added a comment - Patch that handles this. It changes the way we handle rollInterval-based close. Added a couple of tests to test this as well.
          Hide
          Hari Shreedharan added a comment -

          On idle or roll time based close, the bucket writer is closed and the bucket writer is removed from sfWriters map.
          The patch introduces a new exception, which is thrown when an append tries to write to a bucket writer that is closed based on time (roll or idle).

          The sink then catches this exception, creates a new bucket writer and writes to that one.

          Show
          Hari Shreedharan added a comment - On idle or roll time based close, the bucket writer is closed and the bucket writer is removed from sfWriters map. The patch introduces a new exception, which is thrown when an append tries to write to a bucket writer that is closed based on time (roll or idle). The sink then catches this exception, creates a new bucket writer and writes to that one.
          Hide
          Jarek Jarcec Cecho added a comment -

          Thank you for your contribution Hari Shreedharan! I've committed the patch to both trunk and flume-1.5 branch.

          Show
          Jarek Jarcec Cecho added a comment - Thank you for your contribution Hari Shreedharan ! I've committed the patch to both trunk and flume-1.5 branch.
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in flume-trunk #533 (See https://builds.apache.org/job/flume-trunk/533/)
          FLUME-2265. Closed bucket writers should be removed from sfwriters map (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=3b1034e8229eb9ad3e27ed0faab77c3f68f708c6)

          • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
          • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketClosedException.java
          • flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java
          • flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSTestWriterFactory.java
          • flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSBadWriterFactory.java
          • flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java
          • flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSBadSeqWriter.java
          • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
          • flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSTestSeqWriter.java
          Show
          Hudson added a comment - SUCCESS: Integrated in flume-trunk #533 (See https://builds.apache.org/job/flume-trunk/533/ ) FLUME-2265 . Closed bucket writers should be removed from sfwriters map (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=3b1034e8229eb9ad3e27ed0faab77c3f68f708c6 ) flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketClosedException.java flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSTestWriterFactory.java flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSBadWriterFactory.java flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSBadSeqWriter.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSTestSeqWriter.java

            People

            • Assignee:
              Hari Shreedharan
              Reporter:
              Hari Shreedharan
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development