Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2325

BucketWriter might throw BucketClosedException incorrectly

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: v1.4.0
    • Fix Version/s: v1.5.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      in the end of BucketWriter#close(boolean callCloseCallback), variable closed is set to true, which will cause append() to throw BucketClosedException every time when close() is called event if it is a file-size based roll.

      1. FLUME-2325.patch
        0.6 kB
        chenshangan

        Activity

        Hide
        chenshangan521@163.com chenshangan added a comment -

        set variable closed in the following code segment:
        if (callCloseCallback)

        { runCloseAction(); closed = true; }
        Show
        chenshangan521@163.com chenshangan added a comment - set variable closed in the following code segment: if (callCloseCallback) { runCloseAction(); closed = true; }
        Hide
        hshreedharan Hari Shreedharan added a comment -

        +1. I agree with the logic of the patch though I am not entirely sure that this would cause the exception to be thrown since that if(!open) condition in append looks redundant. Roll size and roll count based rolls will trigger a close-open in the if(shouldRotate) block. I don't really see how isOpen can be false and closed be true at the same time in that block. In fact even the closed variable looks redundant as we would remove the reference to this bucket writer immediately after the close.

        Anyway committing this as this is the correct behavior.

        chenshangan - what do you think about removing the closed variables and if(isOpen) ?

        Show
        hshreedharan Hari Shreedharan added a comment - +1. I agree with the logic of the patch though I am not entirely sure that this would cause the exception to be thrown since that if(!open) condition in append looks redundant. Roll size and roll count based rolls will trigger a close-open in the if(shouldRotate) block. I don't really see how isOpen can be false and closed be true at the same time in that block. In fact even the closed variable looks redundant as we would remove the reference to this bucket writer immediately after the close. Anyway committing this as this is the correct behavior. chenshangan - what do you think about removing the closed variables and if(isOpen) ?
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit f32fe79c465d15e813bd54cf1bf5ac59157c1dca in flume's branch refs/heads/trunk from Hari Shreedharan
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=f32fe79 ]

        FLUME-2325. Bucketwriter must be permanently closed only on idle and roll timeouts.

        (chenshangan via Hari Shreedharan)

        Show
        jira-bot ASF subversion and git services added a comment - Commit f32fe79c465d15e813bd54cf1bf5ac59157c1dca in flume's branch refs/heads/trunk from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=f32fe79 ] FLUME-2325 . Bucketwriter must be permanently closed only on idle and roll timeouts. (chenshangan via Hari Shreedharan)
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 3f5db796fe0239e4885cc3ae3233e1224cfb3fb8 in flume's branch refs/heads/flume-1.5 from Hari Shreedharan
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=3f5db79 ]

        FLUME-2325. Bucketwriter must be permanently closed only on idle and roll timeouts.

        (chenshangan via Hari Shreedharan)

        Show
        jira-bot ASF subversion and git services added a comment - Commit 3f5db796fe0239e4885cc3ae3233e1224cfb3fb8 in flume's branch refs/heads/flume-1.5 from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=3f5db79 ] FLUME-2325 . Bucketwriter must be permanently closed only on idle and roll timeouts. (chenshangan via Hari Shreedharan)
        Hide
        hshreedharan Hari Shreedharan added a comment -

        Committed! Thanks chenshangan!

        Show
        hshreedharan Hari Shreedharan added a comment - Committed! Thanks chenshangan!
        Hide
        hudson Hudson added a comment -

        UNSTABLE: Integrated in flume-trunk #551 (See https://builds.apache.org/job/flume-trunk/551/)
        FLUME-2325. Bucketwriter must be permanently closed only on idle and roll timeouts. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=f32fe79c465d15e813bd54cf1bf5ac59157c1dca)

        • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
        Show
        hudson Hudson added a comment - UNSTABLE: Integrated in flume-trunk #551 (See https://builds.apache.org/job/flume-trunk/551/ ) FLUME-2325 . Bucketwriter must be permanently closed only on idle and roll timeouts. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=f32fe79c465d15e813bd54cf1bf5ac59157c1dca ) flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java

          People

          • Assignee:
            chenshangan521@163.com chenshangan
            Reporter:
            chenshangan521@163.com chenshangan
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development