Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2420

HDFSEventSink#stop might throw ConcurrentModificationException

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.5.0
    • Fix Version/s: 1.6.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      There's a chance that bucketWriter callback and HDFSEventSink#stop operate on sfWriter together. So synchronized should apply before operation on sfWriter.

        Activity

        Hide
        hudson Hudson added a comment -

        UNSTABLE: Integrated in Flume-trunk-hbase-98 #15 (See https://builds.apache.org/job/Flume-trunk-hbase-98/15/)
        FLUME-2420. HDFS Bucketwriter must access sfWriters map only within synchronized blocks. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=4e08bf7d38bea365d35a6d391d1507a129cc9ba9)

        • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
        Show
        hudson Hudson added a comment - UNSTABLE: Integrated in Flume-trunk-hbase-98 #15 (See https://builds.apache.org/job/Flume-trunk-hbase-98/15/ ) FLUME-2420 . HDFS Bucketwriter must access sfWriters map only within synchronized blocks. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=4e08bf7d38bea365d35a6d391d1507a129cc9ba9 ) flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
        Hide
        hudson Hudson added a comment -

        UNSTABLE: Integrated in flume-trunk #655 (See https://builds.apache.org/job/flume-trunk/655/)
        FLUME-2420. HDFS Bucketwriter must access sfWriters map only within synchronized blocks. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=4e08bf7d38bea365d35a6d391d1507a129cc9ba9)

        • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
        Show
        hudson Hudson added a comment - UNSTABLE: Integrated in flume-trunk #655 (See https://builds.apache.org/job/flume-trunk/655/ ) FLUME-2420 . HDFS Bucketwriter must access sfWriters map only within synchronized blocks. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=4e08bf7d38bea365d35a6d391d1507a129cc9ba9 ) flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
        Hide
        hshreedharan Hari Shreedharan added a comment -

        Committed! Thanks Chenshangan!

        Show
        hshreedharan Hari Shreedharan added a comment - Committed! Thanks Chenshangan!
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 49f8eb2a3825878a055858e47b73fb92fb40e692 in flume's branch refs/heads/flume-1.6 from Hari Shreedharan
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=49f8eb2 ]

        FLUME-2420. HDFS Bucketwriter must access sfWriters map only within synchronized blocks.

        (chenshangan via Hari)

        Show
        jira-bot ASF subversion and git services added a comment - Commit 49f8eb2a3825878a055858e47b73fb92fb40e692 in flume's branch refs/heads/flume-1.6 from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=49f8eb2 ] FLUME-2420 . HDFS Bucketwriter must access sfWriters map only within synchronized blocks. (chenshangan via Hari)
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 4e08bf7d38bea365d35a6d391d1507a129cc9ba9 in flume's branch refs/heads/trunk from Hari Shreedharan
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=4e08bf7 ]

        FLUME-2420. HDFS Bucketwriter must access sfWriters map only within synchronized blocks.

        (chenshangan via Hari)

        Show
        jira-bot ASF subversion and git services added a comment - Commit 4e08bf7d38bea365d35a6d391d1507a129cc9ba9 in flume's branch refs/heads/trunk from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=4e08bf7 ] FLUME-2420 . HDFS Bucketwriter must access sfWriters map only within synchronized blocks. (chenshangan via Hari)
        Hide
        hshreedharan Hari Shreedharan added a comment - - edited

        +1. Committing this after running tests. I added a synchronized block around the clear call at the end of stop as well to fix possible contention with other threads, though any others calling back to sfWriters should be shutdown by then as the executor is shutdown before that.

        Show
        hshreedharan Hari Shreedharan added a comment - - edited +1. Committing this after running tests. I added a synchronized block around the clear call at the end of stop as well to fix possible contention with other threads, though any others calling back to sfWriters should be shutdown by then as the executor is shutdown before that.
        Hide
        chenshangan521@163.com chenshangan added a comment -

        This happens when we try to close all sinks. Following is the exception stack:

        11 七月 2014 09:33:00,741 ERROR [pool-2-thread-1] (org.apache.flume.node.Application.stopAllSinks:119)  - Error while stopping SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@6435d824 counterGroup:{ name:null counters:{runner.backoffs.consecutive=0, runner.backoffs=224} } }
        java.util.ConcurrentModificationException    at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:394)    at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:413)
            at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:412)
            at org.apache.flume.sink.hdfs.HDFSEventSink.stop(HDFSEventSink.java:615)    at org.apache.flume.sink.DefaultSinkProcessor.stop(DefaultSinkProcessor.java:53)    at org.apache.flume.SinkRunner.stop(SinkRunner.java:122)
            at org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise(LifecycleSupervisor.java:172)
            at org.apache.flume.node.Application.stopAllSinks(Application.java:117)    at org.apache.flume.node.SinkZookeeperRunnable.run(SinkZookeeperRunnable.java:71)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)    
            at java.lang.Thread.run(Thread.java:722)
        
        Show
        chenshangan521@163.com chenshangan added a comment - This happens when we try to close all sinks. Following is the exception stack: 11 七月 2014 09:33:00,741 ERROR [pool-2-thread-1] (org.apache.flume.node.Application.stopAllSinks:119) - Error while stopping SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@6435d824 counterGroup:{ name: null counters:{runner.backoffs.consecutive=0, runner.backoffs=224} } } java.util.ConcurrentModificationException at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:394) at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:413) at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:412) at org.apache.flume.sink.hdfs.HDFSEventSink.stop(HDFSEventSink.java:615) at org.apache.flume.sink.DefaultSinkProcessor.stop(DefaultSinkProcessor.java:53) at org.apache.flume.SinkRunner.stop(SinkRunner.java:122) at org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise(LifecycleSupervisor.java:172) at org.apache.flume.node.Application.stopAllSinks(Application.java:117) at org.apache.flume.node.SinkZookeeperRunnable.run(SinkZookeeperRunnable.java:71) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:722)

          People

          • Assignee:
            chenshangan521@163.com chenshangan
            Reporter:
            chenshangan521@163.com chenshangan
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development