Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4194

ConcurrentModificationError in DirectoryCollection

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.3, 2.0.0-alpha
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: mrv2
    • Labels:
      None

      Description

      As found as part of work on MAPREDUCE-4169, it is possible for a ConcurrentModificationException to be thrown upon disk failure. DirectoryCollection hands out its internal list structure that is accessed across multiple threads. Upon disk failure its internal list is modified, invalidating all current iterators to that structure.

      1. MAPREDUCE-4194.patch
        5 kB
        Jonathan Eagles
      2. MAPREDUCE-4194.patch
        6 kB
        Jonathan Eagles

        Activity

        Hide
        Jonathan Eagles added a comment -

        I have verified that the newly added test fails with ConcurrentModificationException without the changes and succeeds with the changes.

        Show
        Jonathan Eagles added a comment - I have verified that the newly added test fails with ConcurrentModificationException without the changes and succeeds with the changes.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12524066/MAPREDUCE-4194.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.yarn.server.TestContainerManagerSecurity
        org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens
        org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
        org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
        org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
        org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
        org.apache.hadoop.mapred.TestClientRedirect
        org.apache.hadoop.mapreduce.security.TestJHSSecurity

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2303//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2303//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524066/MAPREDUCE-4194.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.TestContainerManagerSecurity org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapreduce.security.TestJHSSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2303//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2303//console This message is automatically generated.
        Hide
        Jonathan Eagles added a comment -

        MAPREDUCE-4189 covers TestContainerManagerSecurity and also shows the other test failures and preexisting.

        Show
        Jonathan Eagles added a comment - MAPREDUCE-4189 covers TestContainerManagerSecurity and also shows the other test failures and preexisting.
        Hide
        Jonathan Eagles added a comment -

        The approach for this bug was to prevent concurrent access modification issues by using a structure that supports versioning of the data. Another approach was to give a copy of the data structure at a cost of performance.

        Show
        Jonathan Eagles added a comment - The approach for this bug was to prevent concurrent access modification issues by using a structure that supports versioning of the data. Another approach was to give a copy of the data structure at a cost of performance.
        Hide
        Robert Joseph Evans added a comment -

        The test will delete the ./target dir, which is not really something that I want removed after running tests because it may have all of my compiled .class files in it, along with who knows what else. I would rather see something like target/<testName>/ as the testDir.

        Also getGoodDirs() and getFailedDirs() are still returning internal data structures. This is not really what this bug was trying to address because this is just to avoid a concurrent modification exception, but I is still bad practice, because I can delete an entry from the list returned and that will be reflected for all people with the list. I would like to see the Lists returned to be immutable, or if they are changed they will have no impact on DirectoryCollection. That can be done on a separate JIRA

        Show
        Robert Joseph Evans added a comment - The test will delete the ./target dir, which is not really something that I want removed after running tests because it may have all of my compiled .class files in it, along with who knows what else. I would rather see something like target/<testName>/ as the testDir. Also getGoodDirs() and getFailedDirs() are still returning internal data structures. This is not really what this bug was trying to address because this is just to avoid a concurrent modification exception, but I is still bad practice, because I can delete an entry from the list returned and that will be reflected for all people with the list. I would like to see the Lists returned to be immutable, or if they are changed they will have no impact on DirectoryCollection. That can be done on a separate JIRA
        Hide
        Jonathan Eagles added a comment -

        Thanks for the review, Bobby. I have changed the test to put temp files in target/<testclass>/* and that dir is now cleaned up after the test class is run. Also I have changed the return type to an unmodifiableList, keeping the CopyOnWriteArrayList backing store. This prevents the ConcurrentModificationException while protecting the internal data structure from being accidentally being changed.

        Note the test is in a new file.

        Show
        Jonathan Eagles added a comment - Thanks for the review, Bobby. I have changed the test to put temp files in target/<testclass>/* and that dir is now cleaned up after the test class is run. Also I have changed the return type to an unmodifiableList, keeping the CopyOnWriteArrayList backing store. This prevents the ConcurrentModificationException while protecting the internal data structure from being accidentally being changed. Note the test is in a new file.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12524324/MAPREDUCE-4194.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.yarn.server.TestContainerManagerSecurity
        org.apache.hadoop.mapreduce.TestYarnClientProtocolProvider

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2309//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2309//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524324/MAPREDUCE-4194.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.TestContainerManagerSecurity org.apache.hadoop.mapreduce.TestYarnClientProtocolProvider +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2309//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2309//console This message is automatically generated.
        Hide
        Jonathan Eagles added a comment -

        Tests above are unrelated.

        Show
        Jonathan Eagles added a comment - Tests above are unrelated.
        Hide
        Robert Joseph Evans added a comment -

        Thanks Jon, I put this into trunk, branch-2, and branch-0.23.

        Show
        Robert Joseph Evans added a comment - Thanks Jon, I put this into trunk, branch-2, and branch-0.23.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2207 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2207/)
        MAPREDUCE-4194. ConcurrentModificationError in DirectoryCollection (Jonathan Eagles via bobby) (Revision 1330552)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330552
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2207 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2207/ ) MAPREDUCE-4194 . ConcurrentModificationError in DirectoryCollection (Jonathan Eagles via bobby) (Revision 1330552) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330552 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java
        Hide
        Jonathan Eagles added a comment -

        Thanks, Bobby!

        Show
        Jonathan Eagles added a comment - Thanks, Bobby!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2133 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2133/)
        MAPREDUCE-4194. ConcurrentModificationError in DirectoryCollection (Jonathan Eagles via bobby) (Revision 1330552)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330552
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2133 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2133/ ) MAPREDUCE-4194 . ConcurrentModificationError in DirectoryCollection (Jonathan Eagles via bobby) (Revision 1330552) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330552 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2150 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2150/)
        MAPREDUCE-4194. ConcurrentModificationError in DirectoryCollection (Jonathan Eagles via bobby) (Revision 1330552)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330552
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2150 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2150/ ) MAPREDUCE-4194 . ConcurrentModificationError in DirectoryCollection (Jonathan Eagles via bobby) (Revision 1330552) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330552 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #239 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/239/)
        svn merge -c 1330552 FIXES: MAPREDUCE-4194. ConcurrentModificationError in DirectoryCollection (Jonathan Eagles via bobby) (Revision 1330556)

        Result = UNSTABLE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330556
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #239 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/239/ ) svn merge -c 1330552 FIXES: MAPREDUCE-4194 . ConcurrentModificationError in DirectoryCollection (Jonathan Eagles via bobby) (Revision 1330556) Result = UNSTABLE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330556 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1026 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1026/)
        MAPREDUCE-4194. ConcurrentModificationError in DirectoryCollection (Jonathan Eagles via bobby) (Revision 1330552)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330552
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1026 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1026/ ) MAPREDUCE-4194 . ConcurrentModificationError in DirectoryCollection (Jonathan Eagles via bobby) (Revision 1330552) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330552 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1061 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1061/)
        MAPREDUCE-4194. ConcurrentModificationError in DirectoryCollection (Jonathan Eagles via bobby) (Revision 1330552)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330552
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1061 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1061/ ) MAPREDUCE-4194 . ConcurrentModificationError in DirectoryCollection (Jonathan Eagles via bobby) (Revision 1330552) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330552 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java

          People

          • Assignee:
            Jonathan Eagles
            Reporter:
            Jonathan Eagles
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development