Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2325

BucketWriter might throw BucketClosedException incorrectly

        Details

        • Type: Bug
        • Status: Resolved
        • Priority: Minor
        • Resolution: Fixed
        • Affects Version/s: 1.4.0
        • Fix Version/s: 1.5.0
        • Component/s: Sinks+Sources
        • Labels:
          None

          Description

          in the end of BucketWriter#close(boolean callCloseCallback), variable closed is set to true, which will cause append() to throw BucketClosedException every time when close() is called event if it is a file-size based roll.

          1. Text File
            FLUME-2325.patch
            0.6 kB
            chenshangan

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            chenshangan521@163.com chenshangan added a comment -

            set variable closed in the following code segment:
            if (callCloseCallback)

            { runCloseAction(); closed = true; }
            Show
            chenshangan521@163.com chenshangan added a comment - set variable closed in the following code segment: if (callCloseCallback) { runCloseAction(); closed = true; }
            Hide
            Permalink
            hshreedharan Hari Shreedharan added a comment -

            +1. I agree with the logic of the patch though I am not entirely sure that this would cause the exception to be thrown since that if(!open) condition in append looks redundant. Roll size and roll count based rolls will trigger a close-open in the if(shouldRotate) block. I don't really see how isOpen can be false and closed be true at the same time in that block. In fact even the closed variable looks redundant as we would remove the reference to this bucket writer immediately after the close.

            Anyway committing this as this is the correct behavior.

            chenshangan - what do you think about removing the closed variables and if(isOpen) ?

            Show
            hshreedharan Hari Shreedharan added a comment - +1. I agree with the logic of the patch though I am not entirely sure that this would cause the exception to be thrown since that if(!open) condition in append looks redundant. Roll size and roll count based rolls will trigger a close-open in the if(shouldRotate) block. I don't really see how isOpen can be false and closed be true at the same time in that block. In fact even the closed variable looks redundant as we would remove the reference to this bucket writer immediately after the close. Anyway committing this as this is the correct behavior. chenshangan - what do you think about removing the closed variables and if(isOpen) ?
            Hide
            Permalink
            jira-bot ASF subversion and git services added a comment -

            Commit f32fe79c465d15e813bd54cf1bf5ac59157c1dca in flume's branch refs/heads/trunk from Hari Shreedharan
            [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=f32fe79 ]

            FLUME-2325. Bucketwriter must be permanently closed only on idle and roll timeouts.

            (chenshangan via Hari Shreedharan)

            Show
            jira-bot ASF subversion and git services added a comment - Commit f32fe79c465d15e813bd54cf1bf5ac59157c1dca in flume's branch refs/heads/trunk from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=f32fe79 ] FLUME-2325 . Bucketwriter must be permanently closed only on idle and roll timeouts. (chenshangan via Hari Shreedharan)
            Hide
            Permalink
            jira-bot ASF subversion and git services added a comment -

            Commit 3f5db796fe0239e4885cc3ae3233e1224cfb3fb8 in flume's branch refs/heads/flume-1.5 from Hari Shreedharan
            [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=3f5db79 ]

            FLUME-2325. Bucketwriter must be permanently closed only on idle and roll timeouts.

            (chenshangan via Hari Shreedharan)

            Show
            jira-bot ASF subversion and git services added a comment - Commit 3f5db796fe0239e4885cc3ae3233e1224cfb3fb8 in flume's branch refs/heads/flume-1.5 from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=3f5db79 ] FLUME-2325 . Bucketwriter must be permanently closed only on idle and roll timeouts. (chenshangan via Hari Shreedharan)
            Hide
            Permalink
            hshreedharan Hari Shreedharan added a comment -

            Committed! Thanks chenshangan!

            Show
            hshreedharan Hari Shreedharan added a comment - Committed! Thanks chenshangan!
            Hide
            Permalink
            hudson Hudson added a comment -

            UNSTABLE: Integrated in flume-trunk #551 (See https://builds.apache.org/job/flume-trunk/551/)
            FLUME-2325. Bucketwriter must be permanently closed only on idle and roll timeouts. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=f32fe79c465d15e813bd54cf1bf5ac59157c1dca)

            • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
            Show
            hudson Hudson added a comment - UNSTABLE: Integrated in flume-trunk #551 (See https://builds.apache.org/job/flume-trunk/551/ ) FLUME-2325 . Bucketwriter must be permanently closed only on idle and roll timeouts. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=f32fe79c465d15e813bd54cf1bf5ac59157c1dca ) flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java

              People

              • Assignee:
                chenshangan521@163.com chenshangan
                Reporter:
                chenshangan521@163.com chenshangan
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development