Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1961

[gridmix3] ConcurrentModificationException when shutting down Gridmix

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: contrib/gridmix
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We observed the following exception occasionally at the end of the Gridmix run:

      Exception in thread "StatsCollectorThread" java.util.ConcurrentModificationException
                at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
                at java.util.AbstractList$Itr.next(AbstractList.java:343)
                at org.apache.hadoop.mapred.gridmix.Statistics$StatCollector.updateAndNotifyClusterStatsListeners(Statistics.java:220)
                at org.apache.hadoop.mapred.gridmix.Statistics$StatCollector.run(Statistics.java:205)
      

        Activity

        Hong Tang created issue -
        Hide
        Hong Tang added a comment -

        It seems that StatsCollectorThread is still active when we call "clusterStatlisteners.clear();" from "Statistics.shutdown()".

        A simple fix is to replace "clusterStatlisteners.clear();" with "lusterStatlisteners = new ArrayList<StatListener<ClusterStats>>()".

        While we are at it, we probably should also fix the same problem with jobStatListeners.

        Show
        Hong Tang added a comment - It seems that StatsCollectorThread is still active when we call "clusterStatlisteners.clear();" from "Statistics.shutdown()". A simple fix is to replace "clusterStatlisteners.clear();" with "lusterStatlisteners = new ArrayList<StatListener<ClusterStats>>()". While we are at it, we probably should also fix the same problem with jobStatListeners.
        Hong Tang made changes -
        Field Original Value New Value
        Assignee Hong Tang [ hong.tang ]
        Hide
        Hong Tang added a comment -

        Trivial patch that uses CopyOnWriteArrayList to avoid concurrent modification. No unit test included as it is hard to reproduce the synchronization bug through unit tests.

        Show
        Hong Tang added a comment - Trivial patch that uses CopyOnWriteArrayList to avoid concurrent modification. No unit test included as it is hard to reproduce the synchronization bug through unit tests.
        Hong Tang made changes -
        Attachment mr-1961-20100723.patch [ 12450368 ]
        Hong Tang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hong Tang made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Hong Tang added a comment -

        retry hudson

        Show
        Hong Tang added a comment - retry hudson
        Hong Tang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Ranjit Mathew made changes -
        Component/s contrib/gridmix [ 12313086 ]
        Hide
        Ranjit Mathew added a comment -

        +1 for the patch.

        Show
        Ranjit Mathew added a comment - +1 for the patch.
        Hide
        Hong Tang added a comment -

        test-patch output:

             [exec] -1 overall.
             [exec]
             [exec]     +1 @author.  The patch does not contain any @author tags.
             [exec]
             [exec]     -1 tests included.  The patch doesn't appear to include any new or modified tests.
             [exec]                         Please justify why no new tests are needed for this patch.
             [exec]                         Also please list what manual steps were performed to verify this patch.
             [exec]
             [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
             [exec]
             [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
             [exec]
             [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
             [exec]
             [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
        

        -1 tests included

        The patch fixes a concurrency bug, and is hard to write unit test to cover. The patch is tested with real benchmark runs and has shown to be effective.

        Show
        Hong Tang added a comment - test-patch output: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no new tests are needed for this patch. [exec] Also please list what manual steps were performed to verify this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 tests included The patch fixes a concurrency bug, and is hard to write unit test to cover. The patch is tested with real benchmark runs and has shown to be effective.
        Hide
        Chris Douglas added a comment -

        +1

        I committed this. Thanks, Hong!

        Show
        Chris Douglas added a comment - +1 I committed this. Thanks, Hong!
        Chris Douglas made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Fix Version/s 0.22.0 [ 12314184 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/ )
        Konstantin Shvachko made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Hong Tang
            Reporter:
            Hong Tang
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development