Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2586

HDFS Sink should have an option to try rename even if close fails

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.0
    • Component/s: None
    • Labels:
      None

      Description

      "The issue is that the file close fails, but the stream on the Flume side is considered closed by HDFS. So retrying the close, is failing - which causes the rename never to be attempted. Since the rename is never attempted the file sticks around with the ".tmp" file name. "
      We should try attempting rename even if the close fails and possibly log information that the user need to run lease recovery

        Issue Links

          Activity

          Hide
          jrufus Johny Rufus added a comment -

          Calling close multiple times on the HDFSStream has no effect, hence in this patch, the closeTries parameter is used to retry rename, (irrespective of wether the close succeeded or not)

          Show
          jrufus Johny Rufus added a comment - Calling close multiple times on the HDFSStream has no effect, hence in this patch, the closeTries parameter is used to retry rename, (irrespective of wether the close succeeded or not)
          Hide
          hshreedharan Hari Shreedharan added a comment -

          Johny Rufus, The patch looks good. I am running tests. If all tests pass, I will commit this.

          Show
          hshreedharan Hari Shreedharan added a comment - Johny Rufus , The patch looks good. I am running tests. If all tests pass, I will commit this.
          Hide
          hshreedharan Hari Shreedharan added a comment -

          +1. Committing.

          Show
          hshreedharan Hari Shreedharan added a comment - +1. Committing.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 0d6eccad2781884fe0f000f74b8f964cbdd7971f in flume's branch refs/heads/trunk from Hari Shreedharan
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=0d6ecca ]

          FLUME-2586. HDFS Sink must try to rename files even if close fails.

          (Johny Rufus via Hari)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 0d6eccad2781884fe0f000f74b8f964cbdd7971f in flume's branch refs/heads/trunk from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=0d6ecca ] FLUME-2586 . HDFS Sink must try to rename files even if close fails. (Johny Rufus via Hari)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 13a03003e7c1817d73451d3b6b1a58250c9469bd in flume's branch refs/heads/flume-1.6 from Hari Shreedharan
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=13a0300 ]

          FLUME-2586. HDFS Sink must try to rename files even if close fails.

          (Johny Rufus via Hari)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 13a03003e7c1817d73451d3b6b1a58250c9469bd in flume's branch refs/heads/flume-1.6 from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=13a0300 ] FLUME-2586 . HDFS Sink must try to rename files even if close fails. (Johny Rufus via Hari)
          Hide
          hshreedharan Hari Shreedharan added a comment -

          Committed! Thanks Johny!

          Show
          hshreedharan Hari Shreedharan added a comment - Committed! Thanks Johny!
          Hide
          hudson Hudson added a comment -

          UNSTABLE: Integrated in flume-trunk #709 (See https://builds.apache.org/job/flume-trunk/709/)
          FLUME-2586. HDFS Sink must try to rename files even if close fails. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=0d6eccad2781884fe0f000f74b8f964cbdd7971f)

          • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
          • flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockFsDataOutputStream.java
          • flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java
          • flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java
          • flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockFileSystem.java
          Show
          hudson Hudson added a comment - UNSTABLE: Integrated in flume-trunk #709 (See https://builds.apache.org/job/flume-trunk/709/ ) FLUME-2586 . HDFS Sink must try to rename files even if close fails. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=0d6eccad2781884fe0f000f74b8f964cbdd7971f ) flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockFsDataOutputStream.java flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockFileSystem.java
          Hide
          hudson Hudson added a comment -

          UNSTABLE: Integrated in Flume-trunk-hbase-98 #66 (See https://builds.apache.org/job/Flume-trunk-hbase-98/66/)
          FLUME-2586. HDFS Sink must try to rename files even if close fails. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=0d6eccad2781884fe0f000f74b8f964cbdd7971f)

          • flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockFsDataOutputStream.java
          • flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockFileSystem.java
          • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
          • flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java
          • flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java
          Show
          hudson Hudson added a comment - UNSTABLE: Integrated in Flume-trunk-hbase-98 #66 (See https://builds.apache.org/job/Flume-trunk-hbase-98/66/ ) FLUME-2586 . HDFS Sink must try to rename files even if close fails. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=0d6eccad2781884fe0f000f74b8f964cbdd7971f ) flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockFsDataOutputStream.java flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockFileSystem.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java
          Hide
          roshan_naik Roshan Naik added a comment -

          Johny Rufus Hari Shreedharan looks like this feature went undocumented.

          Show
          roshan_naik Roshan Naik added a comment - Johny Rufus Hari Shreedharan looks like this feature went undocumented.
          Hide
          hshreedharan Hari Shreedharan added a comment -

          It just reuses an older config param, "hdfs.closeTries" - we just need to update the description I guess.

          Show
          hshreedharan Hari Shreedharan added a comment - It just reuses an older config param, "hdfs.closeTries" - we just need to update the description I guess.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 4188e4dedb4af0792bfac3086e3b2a7d415f2248 in flume's branch refs/heads/trunk from Hari Shreedharan
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=4188e4d ]

          FLUME-2665. Update documentation for hdfs.closeTries based on FLUME-2586

          (Johny Rufus via Hari)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 4188e4dedb4af0792bfac3086e3b2a7d415f2248 in flume's branch refs/heads/trunk from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=4188e4d ] FLUME-2665 . Update documentation for hdfs.closeTries based on FLUME-2586 (Johny Rufus via Hari)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit dbf4828b89a36fda9434163f1835382e2f0c56ba in flume's branch refs/heads/flume-1.6 from Hari Shreedharan
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=dbf4828 ]

          FLUME-2665. Update documentation for hdfs.closeTries based on FLUME-2586

          (Johny Rufus via Hari)

          Show
          jira-bot ASF subversion and git services added a comment - Commit dbf4828b89a36fda9434163f1835382e2f0c56ba in flume's branch refs/heads/flume-1.6 from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=dbf4828 ] FLUME-2665 . Update documentation for hdfs.closeTries based on FLUME-2586 (Johny Rufus via Hari)
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Flume-trunk-hbase-98 #90 (See https://builds.apache.org/job/Flume-trunk-hbase-98/90/)
          FLUME-2665. Update documentation for hdfs.closeTries based on FLUME-2586 (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=4188e4dedb4af0792bfac3086e3b2a7d415f2248)

          • flume-ng-doc/sphinx/FlumeUserGuide.rst
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Flume-trunk-hbase-98 #90 (See https://builds.apache.org/job/Flume-trunk-hbase-98/90/ ) FLUME-2665 . Update documentation for hdfs.closeTries based on FLUME-2586 (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=4188e4dedb4af0792bfac3086e3b2a7d415f2248 ) flume-ng-doc/sphinx/FlumeUserGuide.rst

            People

            • Assignee:
              jrufus Johny Rufus
              Reporter:
              jrufus Johny Rufus
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development