Flume
  1. Flume
  2. FLUME-2357

HDFS sink should retry closing files that previously had close errors

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v1.4.0
    • Fix Version/s: v1.5.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      When the AbstractHDFSWriter fails to close a file (due to exceeding the callTimeout or other hdfs issues), it will leave the the file open and never try again. The only way to close the open files is to restart the flume agent. There should be a configurable option to allow the sink to retry to close files that had previously failed to close.

      1. FLUME-2357.patch
        55 kB
        Hari Shreedharan
      2. FLUME-2357-1.patch
        62 kB
        Hari Shreedharan
      3. FLUME-2357-2.patch
        62 kB
        Hari Shreedharan
      4. FLUME-2357-3.patch
        62 kB
        Hari Shreedharan

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          SUCCESS: Integrated in flume-trunk #632 (See https://builds.apache.org/job/flume-trunk/632/)
          FLUME-2357. HDFS sink should retry closing files that previously had close errors (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo/?p=flume.git&a=commit&h=a94594dd2c5cb980bc6f82b1fa606a922986569e)

          • flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockFsDataOutputStreamCloseRetryWrapper.java
          • flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockFsDataOutputStream.java
          • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java
          • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java
          • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java
          • flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestUseRawLocalFileSystem.java
          • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
          • flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockFileSystemCloseRetryWrapper.java
          • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
          • flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockFileSystem.java
          • flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockDataStream.java
          • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java
          • flume-ng-doc/sphinx/FlumeUserGuide.rst
          • flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java
          • flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java
          Show
          Hudson added a comment - SUCCESS: Integrated in flume-trunk #632 (See https://builds.apache.org/job/flume-trunk/632/ ) FLUME-2357 . HDFS sink should retry closing files that previously had close errors (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo/?p=flume.git&a=commit&h=a94594dd2c5cb980bc6f82b1fa606a922986569e ) flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockFsDataOutputStreamCloseRetryWrapper.java flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockFsDataOutputStream.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestUseRawLocalFileSystem.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockFileSystemCloseRetryWrapper.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockFileSystem.java flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockDataStream.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java flume-ng-doc/sphinx/FlumeUserGuide.rst flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java
          Hide
          Mike Percy added a comment -

          Pushed to trunk & flume-1.5 branches.

          Thanks for the patch Hari!

          Show
          Mike Percy added a comment - Pushed to trunk & flume-1.5 branches. Thanks for the patch Hari!
          Hide
          ASF subversion and git services added a comment -

          Commit e984ffa7e86f983953eeaf6fa3f0de3d699dc73b in flume's branch refs/heads/flume-1.5 from Mike Percy
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=e984ffa ]

          FLUME-2357. HDFS sink should retry closing files that previously had close errors

          (Hari Shreedharan via Mike Percy)

          Show
          ASF subversion and git services added a comment - Commit e984ffa7e86f983953eeaf6fa3f0de3d699dc73b in flume's branch refs/heads/flume-1.5 from Mike Percy [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=e984ffa ] FLUME-2357 . HDFS sink should retry closing files that previously had close errors (Hari Shreedharan via Mike Percy)
          Hide
          Mike Percy added a comment -

          +1

          Show
          Mike Percy added a comment - +1
          Hide
          ASF subversion and git services added a comment -

          Commit a94594dd2c5cb980bc6f82b1fa606a922986569e in flume's branch refs/heads/trunk from Mike Percy
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=a94594d ]

          FLUME-2357. HDFS sink should retry closing files that previously had close errors

          (Hari Shreedharan via Mike Percy)

          Show
          ASF subversion and git services added a comment - Commit a94594dd2c5cb980bc6f82b1fa606a922986569e in flume's branch refs/heads/trunk from Mike Percy [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=a94594d ] FLUME-2357 . HDFS sink should retry closing files that previously had close errors (Hari Shreedharan via Mike Percy)
          Hide
          Hari Shreedharan added a comment -

          Added a couple of bad configuration tests. Also added a test in HDFS sink to test for retries configured via the configuration

          Show
          Hari Shreedharan added a comment - Added a couple of bad configuration tests. Also added a test in HDFS sink to test for retries configured via the configuration
          Hide
          Hari Shreedharan added a comment -

          Much of the size of the patch is due to a couple of file renames. Otherwise the patch itself is pretty simple. In the Bucketwriter, if a close fails, we simply reschedule the close to happen sometime later until it finally succeeds or till we hit a maximum count. Added a test case too. This depends on the presence of the isFileClosed method in the HDFS client API. If the method is absent, reattempts are not done.

          Show
          Hari Shreedharan added a comment - Much of the size of the patch is due to a couple of file renames. Otherwise the patch itself is pretty simple. In the Bucketwriter, if a close fails, we simply reschedule the close to happen sometime later until it finally succeeds or till we hit a maximum count. Added a test case too. This depends on the presence of the isFileClosed method in the HDFS client API. If the method is absent, reattempts are not done.

            People

            • Assignee:
              Hari Shreedharan
              Reporter:
              Patrick Dvorak
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development