Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-606

ConcurrentModificationException in invalidateCorruptReplicas()

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20-append, 0.20.205.0, 0.21.0
    • Fix Version/s: 0.20-append, 0.20.205.0, 0.21.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      BlockManager.invalidateCorruptReplicas() iterates over DatanodeDescriptor-s while removing corrupt replicas from the descriptors. This causes ConcurrentModificationException if there is more than one replicas of the block. I ran into this exception debugging different scenarios in append, but it should be fixed in the trunk too.

      1. CMEinCorruptReplicas.patch
        0.9 kB
        Konstantin Shvachko
      2. hdfs-606-20s.patch
        3 kB
        Suresh Srinivas
      3. hdfs-606-branch20.txt
        4 kB
        Todd Lipcon

        Issue Links

          Activity

          Konstantin Shvachko created issue -
          Hide
          Konstantin Shvachko added a comment -

          This is a rather obvious patch, which makes a copy of the list before iterating.
          Should not require any new tests, because this condition is hardly testable at all.

          Show
          Konstantin Shvachko added a comment - This is a rather obvious patch, which makes a copy of the list before iterating. Should not require any new tests, because this condition is hardly testable at all.
          Konstantin Shvachko made changes -
          Field Original Value New Value
          Attachment CMEinCorruptReplicas.patch [ 12419013 ]
          Konstantin Shvachko made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12419013/CMEinCorruptReplicas.patch
          against trunk revision 812701.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/18/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/18/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/18/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/18/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419013/CMEinCorruptReplicas.patch against trunk revision 812701. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/18/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/18/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/18/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/18/console This message is automatically generated.
          Konstantin Shvachko made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Konstantin Shvachko added a comment -

          Resubmitting because none of the three tests fail on my box. Trying to see whether the failures from different Hudson runs will be consistent.

          Show
          Konstantin Shvachko added a comment - Resubmitting because none of the three tests fail on my box. Trying to see whether the failures from different Hudson runs will be consistent.
          Konstantin Shvachko made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12419013/CMEinCorruptReplicas.patch
          against trunk revision 812701.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/19/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/19/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/19/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/19/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419013/CMEinCorruptReplicas.patch against trunk revision 812701. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/19/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/19/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/19/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/19/console This message is automatically generated.
          Hide
          Jakob Homan added a comment -

          +1. Looks good.

          Show
          Jakob Homan added a comment - +1. Looks good.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #29 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/29/)
          . Fix ConcurrentModificationException in invalidateCorruptReplicas(). Contributed by Konstantin Shvachko.

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #29 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/29/ ) . Fix ConcurrentModificationException in invalidateCorruptReplicas(). Contributed by Konstantin Shvachko.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #81 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/81/)
          . Fix ConcurrentModificationException in invalidateCorruptReplicas(). Contributed by Konstantin Shvachko.

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #81 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/81/ ) . Fix ConcurrentModificationException in invalidateCorruptReplicas(). Contributed by Konstantin Shvachko.
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #26 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/26/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #26 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/26/ )
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #6 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/6/)
          . Fix ConcurrentModificationException in invalidateCorruptReplicas(). Contributed by Konstantin Shvachko.

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #6 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/6/ ) . Fix ConcurrentModificationException in invalidateCorruptReplicas(). Contributed by Konstantin Shvachko.
          Hide
          Jakob Homan added a comment -

          This was committed to trunk Fri Sep 11 21:54:24 UTC 2009. Resolving as fixed.

          Show
          Jakob Homan added a comment - This was committed to trunk Fri Sep 11 21:54:24 UTC 2009. Resolving as fixed.
          Jakob Homan made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]
          Hide
          Todd Lipcon added a comment -

          Ran into this issue stress testing the append work on branch 20. This patch includes a test case that demonstrates the issue (test case depends on some of the test utils from other branch 20 append patches)

          Show
          Todd Lipcon added a comment - Ran into this issue stress testing the append work on branch 20. This patch includes a test case that demonstrates the issue (test case depends on some of the test utils from other branch 20 append patches)
          Todd Lipcon made changes -
          Attachment hdfs-606-branch20.txt [ 12444260 ]
          Todd Lipcon made changes -
          Link This issue blocks HDFS-142 [ HDFS-142 ]
          Hide
          Nicolas Spiegelberg added a comment -

          This should be pulled into the branch-0.20-append branch.

          Show
          Nicolas Spiegelberg added a comment - This should be pulled into the branch-0.20-append branch.
          Nicolas Spiegelberg made changes -
          Affects Version/s 0.20-append [ 12315103 ]
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Suresh Srinivas made changes -
          Fix Version/s 0.20-append [ 12315103 ]
          Fix Version/s 0.20.205.0 [ 12316392 ]
          Affects Version/s 0.20.205.0 [ 12316392 ]
          Suresh Srinivas made changes -
          Attachment hdfs-606-20s.patch [ 12491261 ]
          Suresh Srinivas made changes -
          Attachment hdfs-606-20s.patch [ 12491261 ]
          Hide
          Suresh Srinivas added a comment -

          Merged the patch from 0.20-append to 0.20-security. It fixes concurrent modification exception. Test changes are not ported because it has dependency on other changes from 0.20-append.

          Show
          Suresh Srinivas added a comment - Merged the patch from 0.20-append to 0.20-security. It fixes concurrent modification exception. Test changes are not ported because it has dependency on other changes from 0.20-append.
          Suresh Srinivas made changes -
          Attachment hdfs-606-20s.patch [ 12491262 ]
          Hide
          Jitendra Nath Pandey added a comment -

          +1 for hdfs-606-20s.patch.

          Show
          Jitendra Nath Pandey added a comment - +1 for hdfs-606-20s.patch.
          Gavin made changes -
          Link This issue blocks HDFS-142 [ HDFS-142 ]
          Gavin made changes -
          Link This issue is depended upon by HDFS-142 [ HDFS-142 ]

            People

            • Assignee:
              Konstantin Shvachko
              Reporter:
              Konstantin Shvachko
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development