Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4226

ConcurrentModificationException in FileSystemCounterGroup

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.1
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: mrv2
    • Labels:
      None

      Description

      This was seen in a Hive job. I'll attach a failing test case.

        Activity

        Tom White created issue -
        Hide
        Tom White added a comment -

        Here's a unit test to reproduce the problem, and a fix. The fix uses a ConcurrentSkipListMap like we did in MAPREDUCE-3749/MAPREDUCE-3827.

        Show
        Tom White added a comment - Here's a unit test to reproduce the problem, and a fix. The fix uses a ConcurrentSkipListMap like we did in MAPREDUCE-3749 / MAPREDUCE-3827 .
        Tom White made changes -
        Field Original Value New Value
        Attachment MAPREDUCE-4226.patch [ 12525688 ]
        Tom White made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Target Version/s 2.0.0 [ 12320354 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12525688/MAPREDUCE-4226.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2359//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2359//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525688/MAPREDUCE-4226.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2359//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2359//console This message is automatically generated.
        Hide
        Alejandro Abdelnur added a comment -

        +1. lgtm

        Show
        Alejandro Abdelnur added a comment - +1. lgtm
        Hide
        Robert Joseph Evans added a comment -

        The patch looks good to me too. I see one small issue that was already in the code before the patch, but I would like to see fixed. Write(DataOutputStream) is not synchronized. I assume that this is not a big deal because it can only be a problem if we are writing out the counters at the same time we are adding in new counters. I don't think that happens very often, but if it does the number of schemes or the number of counters in a scheme could be off. This could result in exceptions on the reader side, or possibly some data corruption in the counters on the reader side.

        I also would like to see this fix go into branch-0.23 (The patch appears to apply cleanly with no issues).

        Show
        Robert Joseph Evans added a comment - The patch looks good to me too. I see one small issue that was already in the code before the patch, but I would like to see fixed. Write(DataOutputStream) is not synchronized. I assume that this is not a big deal because it can only be a problem if we are writing out the counters at the same time we are adding in new counters. I don't think that happens very often, but if it does the number of schemes or the number of counters in a scheme could be off. This could result in exceptions on the reader side, or possibly some data corruption in the counters on the reader side. I also would like to see this fix go into branch-0.23 (The patch appears to apply cleanly with no issues).
        Hide
        Tom White added a comment -

        I'm not sure if this is a problem in practice since counters are all written together, and o.a.h.mapreduce.counters.AbstractCounters#write() is synchronized, as is findCounter() which is used to access and increment framework counters (e.g. in FileSystemStatisticUpdater). So I think we can commit this as an incremental improvement, and follow up with another JIRA if there does turn out to be a problem with synchronization on FileSystemCounterGroup or FrameworkCounterGroup. Does that sound reasonable?

        Show
        Tom White added a comment - I'm not sure if this is a problem in practice since counters are all written together, and o.a.h.mapreduce.counters.AbstractCounters#write() is synchronized, as is findCounter() which is used to access and increment framework counters (e.g. in FileSystemStatisticUpdater). So I think we can commit this as an incremental improvement, and follow up with another JIRA if there does turn out to be a problem with synchronization on FileSystemCounterGroup or FrameworkCounterGroup. Does that sound reasonable?
        Hide
        Robert Joseph Evans added a comment -

        I wasn't sure if it was a critical issue or not, because I didn't have much context with how this class was used. I am +1 for putting in the patch unchanged. I still think it needs to be fixed, mostly as defensive programming so that if things do change in the future this issue does not crop up. Especially because it is just a single line change that should not really impact the performance as it is only being called from something that is holding a bigger lock.

        Show
        Robert Joseph Evans added a comment - I wasn't sure if it was a critical issue or not, because I didn't have much context with how this class was used. I am +1 for putting in the patch unchanged. I still think it needs to be fixed, mostly as defensive programming so that if things do change in the future this issue does not crop up. Especially because it is just a single line change that should not really impact the performance as it is only being called from something that is holding a bigger lock.
        Hide
        Tom White added a comment -

        Robert, I opened MAPREDUCE-4230 for the follow-on issue. I've just committed this (to the 0.23 and 2 branches).

        Show
        Tom White added a comment - Robert, I opened MAPREDUCE-4230 for the follow-on issue. I've just committed this (to the 0.23 and 2 branches).
        Tom White made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 0.23.3 [ 12320060 ]
        Fix Version/s 2.0.0 [ 12320354 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2198 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2198/)
        MAPREDUCE-4226. ConcurrentModificationException in FileSystemCounterGroup. (Revision 1335197)

        Result = SUCCESS
        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335197
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2198 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2198/ ) MAPREDUCE-4226 . ConcurrentModificationException in FileSystemCounterGroup. (Revision 1335197) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335197 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2273 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2273/)
        MAPREDUCE-4226. ConcurrentModificationException in FileSystemCounterGroup. (Revision 1335197)

        Result = SUCCESS
        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335197
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2273 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2273/ ) MAPREDUCE-4226 . ConcurrentModificationException in FileSystemCounterGroup. (Revision 1335197) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335197 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2215 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2215/)
        MAPREDUCE-4226. ConcurrentModificationException in FileSystemCounterGroup. (Revision 1335197)

        Result = ABORTED
        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335197
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2215 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2215/ ) MAPREDUCE-4226 . ConcurrentModificationException in FileSystemCounterGroup. (Revision 1335197) Result = ABORTED tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335197 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #251 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/251/)
        Merge -r 1335196:1335197 from trunk to branch-0.23. Fixes: MAPREDUCE-4226 (Revision 1335199)

        Result = SUCCESS
        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335199
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #251 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/251/ ) Merge -r 1335196:1335197 from trunk to branch-0.23. Fixes: MAPREDUCE-4226 (Revision 1335199) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335199 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1038 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1038/)
        MAPREDUCE-4226. ConcurrentModificationException in FileSystemCounterGroup. (Revision 1335197)

        Result = FAILURE
        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335197
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1038 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1038/ ) MAPREDUCE-4226 . ConcurrentModificationException in FileSystemCounterGroup. (Revision 1335197) Result = FAILURE tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335197 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1073 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1073/)
        MAPREDUCE-4226. ConcurrentModificationException in FileSystemCounterGroup. (Revision 1335197)

        Result = SUCCESS
        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335197
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1073 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1073/ ) MAPREDUCE-4226 . ConcurrentModificationException in FileSystemCounterGroup. (Revision 1335197) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335197 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Arun C Murthy made changes -
        Fix Version/s 2.0.2-alpha [ 12322471 ]
        Fix Version/s 2.0.0-alpha [ 12320354 ]

          People

          • Assignee:
            Tom White
            Reporter:
            Tom White
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development