Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3005

ConcurrentModificationException in FSDataset$FSVolume.getDfsUsed(..)

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.0-alpha
    • Component/s: datanode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Saw this in build #1888.

      java.util.ConcurrentModificationException
      	at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
      	at java.util.HashMap$EntryIterator.next(HashMap.java:834)
      	at java.util.HashMap$EntryIterator.next(HashMap.java:832)
      	at org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.getDfsUsed(FSDataset.java:557)
      	at org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolumeSet.getDfsUsed(FSDataset.java:809)
      	at org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolumeSet.access$1400(FSDataset.java:774)
      	at org.apache.hadoop.hdfs.server.datanode.FSDataset.getDfsUsed(FSDataset.java:1124)
      	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.sendHeartBeat(BPOfferService.java:406)
      	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.offerService(BPOfferService.java:490)
      	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.run(BPOfferService.java:635)
      	at java.lang.Thread.run(Thread.java:662)
      
      1. h3005_20120314b.patch
        10 kB
        Tsz Wo Nicholas Sze
      2. h3005_20120314.patch
        10 kB
        Tsz Wo Nicholas Sze
      3. h3005_20120312.patch
        7 kB
        Tsz Wo Nicholas Sze
      4. HDFS-3005.patch
        1 kB
        Vinayakumar B

        Activity

        Allen Wittenauer made changes -
        Affects Version/s 2.0.0-alpha [ 12320353 ]
        Affects Version/s 0.24.0 [ 12317653 ]
        Arun C Murthy made changes -
        Fix Version/s 2.0.0 [ 12320353 ]
        Fix Version/s 0.24.0 [ 12317653 ]
        Fix Version/s 0.23.3 [ 12320052 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1021 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1021/)
        HDFS-3005. FSVolume.decDfsUsed(..) should be synchronized. (Revision 1301127)

        Result = FAILURE
        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301127
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1021 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1021/ ) HDFS-3005 . FSVolume.decDfsUsed(..) should be synchronized. (Revision 1301127) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301127 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #227 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/227/)
        svn merge -c 1301127 from trunk for HDFS-3005. (Revision 1301130)

        Result = FAILURE
        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301130
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #227 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/227/ ) svn merge -c 1301127 from trunk for HDFS-3005 . (Revision 1301130) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301130 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #199 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/199/)
        svn merge -c 1301127 from trunk for HDFS-3005. (Revision 1301130)

        Result = SUCCESS
        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301130
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #199 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/199/ ) svn merge -c 1301127 from trunk for HDFS-3005 . (Revision 1301130) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301130 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #986 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/986/)
        HDFS-3005. FSVolume.decDfsUsed(..) should be synchronized. (Revision 1301127)

        Result = SUCCESS
        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301127
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #986 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/986/ ) HDFS-3005 . FSVolume.decDfsUsed(..) should be synchronized. (Revision 1301127) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301127 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Commit #692 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/692/)
        svn merge -c 1301127 from trunk for HDFS-3005. (Revision 1301130)

        Result = ABORTED
        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301130
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #692 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/692/ ) svn merge -c 1301127 from trunk for HDFS-3005 . (Revision 1301130) Result = ABORTED szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301130 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1887 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1887/)
        HDFS-3005. FSVolume.decDfsUsed(..) should be synchronized. (Revision 1301127)

        Result = ABORTED
        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301127
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1887 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1887/ ) HDFS-3005 . FSVolume.decDfsUsed(..) should be synchronized. (Revision 1301127) Result = ABORTED szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301127 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1953 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1953/)
        HDFS-3005. FSVolume.decDfsUsed(..) should be synchronized. (Revision 1301127)

        Result = SUCCESS
        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301127
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1953 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1953/ ) HDFS-3005 . FSVolume.decDfsUsed(..) should be synchronized. (Revision 1301127) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301127 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Commit #675 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/675/)
        svn merge -c 1301127 from trunk for HDFS-3005. (Revision 1301130)

        Result = SUCCESS
        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301130
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #675 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/675/ ) svn merge -c 1301127 from trunk for HDFS-3005 . (Revision 1301130) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301130 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-0.23-Commit #684 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/684/)
        svn merge -c 1301127 from trunk for HDFS-3005. (Revision 1301130)

        Result = SUCCESS
        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301130
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #684 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/684/ ) svn merge -c 1301127 from trunk for HDFS-3005 . (Revision 1301130) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301130 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1878 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1878/)
        HDFS-3005. FSVolume.decDfsUsed(..) should be synchronized. (Revision 1301127)

        Result = SUCCESS
        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301127
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1878 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1878/ ) HDFS-3005 . FSVolume.decDfsUsed(..) should be synchronized. (Revision 1301127) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301127 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
        Tsz Wo Nicholas Sze made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 0.24.0 [ 12317653 ]
        Fix Version/s 0.23.3 [ 12320052 ]
        Resolution Fixed [ 1 ]
        Hide
        Tsz Wo Nicholas Sze added a comment -

        I have committed this to trunk and 0.23.

        Show
        Tsz Wo Nicholas Sze added a comment - I have committed this to trunk and 0.23.
        Hide
        Suresh Srinivas added a comment -

        +1 for the change

        Show
        Suresh Srinivas added a comment - +1 for the change
        Tsz Wo Nicholas Sze made changes -
        Attachment h3005_20120314b.patch [ 12518408 ]
        Hide
        Tsz Wo Nicholas Sze added a comment -

        > FSVolume is synchronized by FSDataset. However FSVolume#checkDirs() is synchronized by FSVolumeset. ...

        Oops, I missed it, good catch.

        h3005_20120314b.patch: add the TODO back to checkDir(). No code change.

        Show
        Tsz Wo Nicholas Sze added a comment - > FSVolume is synchronized by FSDataset. However FSVolume#checkDirs() is synchronized by FSVolumeset. ... Oops, I missed it, good catch. h3005_20120314b.patch: add the TODO back to checkDir(). No code change.
        Hide
        Suresh Srinivas added a comment -

        FSVolume is synchronized by FSDataset. However FSVolume#checkDirs() is synchronized by FSVolumeset. So either we fix that issue in this jira or leave the TODO back and fix it in another jira.

        Show
        Suresh Srinivas added a comment - FSVolume is synchronized by FSDataset. However FSVolume#checkDirs() is synchronized by FSVolumeset. So either we fix that issue in this jira or leave the TODO back and fix it in another jira.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12518369/h3005_20120314.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.cli.TestHDFSCLI

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2002//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2002//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12518369/h3005_20120314.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.cli.TestHDFSCLI +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2002//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2002//console This message is automatically generated.
        Tsz Wo Nicholas Sze made changes -
        Attachment h3005_20120314.patch [ 12518369 ]
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Suresh, thanks for the review.

        • Since we are seeing ConcurrentModificationException on existing tests, the existing tests already cover it. We can add a new test for this specific condition but it is not very useful because we already knew the problem.
        • I will move the inner classes out in a subtask of HDFS-3080.

        h3005_20120314.patch: address all other comments.

        Show
        Tsz Wo Nicholas Sze added a comment - Suresh, thanks for the review. Since we are seeing ConcurrentModificationException on existing tests, the existing tests already cover it. We can add a new test for this specific condition but it is not very useful because we already knew the problem. I will move the inner classes out in a subtask of HDFS-3080 . h3005_20120314.patch: address all other comments.
        Hide
        Suresh Srinivas added a comment -

        Comments:

        1. FSDir has line > 80 columns
        2. decDfsUsed() - can you add to the method javadoc, that the call to this method is made only from FSVolume#decDfsUsed(), which is synchronized.
        3. Please add to FSVolume class javadoc, it uses the FSDataset for synchronization. Also add javadoc to BlockPoolSlice class that it does not have any synchronization. It is used only by FSVolume and is synchronized by it.
        4. Can you also handle checkDirs() TODO:
        5. We should consider moving the static classes into a separate file. Also we should consider making BlockPoolSlice an inner class of FSVolume. This could be in another jira.

        I am not sure how we can test this. Any ideas?

        Show
        Suresh Srinivas added a comment - Comments: FSDir has line > 80 columns decDfsUsed() - can you add to the method javadoc, that the call to this method is made only from FSVolume#decDfsUsed(), which is synchronized. Please add to FSVolume class javadoc, it uses the FSDataset for synchronization. Also add javadoc to BlockPoolSlice class that it does not have any synchronization. It is used only by FSVolume and is synchronized by it. Can you also handle checkDirs() TODO: We should consider moving the static classes into a separate file. Also we should consider making BlockPoolSlice an inner class of FSVolume. This could be in another jira. I am not sure how we can test this. Any ideas?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12518144/h3005_20120312.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.cli.TestHDFSCLI

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1998//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1998//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12518144/h3005_20120312.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.cli.TestHDFSCLI +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1998//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1998//console This message is automatically generated.
        Tsz Wo Nicholas Sze made changes -
        Assignee Tsz Wo (Nicholas), SZE [ szetszwo ]
        Tsz Wo Nicholas Sze made changes -
        Attachment h3005_20120312.patch [ 12518144 ]
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Hi VINAYAKUMAR,

        Thanks for posting a patch. ConcurrentHashMap won't work since the map is being iterated.

        h3005_20120312.patch: synchronizes dataset and changes the inner FSDataset classes to static.

        Show
        Tsz Wo Nicholas Sze added a comment - Hi VINAYAKUMAR, Thanks for posting a patch. ConcurrentHashMap won't work since the map is being iterated. h3005_20120312.patch: synchronizes dataset and changes the inner FSDataset classes to static.
        Vinayakumar B made changes -
        Attachment HDFS-3005.patch [ 12517889 ]
        Hide
        Vinayakumar B added a comment -

        Attaching the Latest Patch.

        Show
        Vinayakumar B added a comment - Attaching the Latest Patch.
        Vinayakumar B made changes -
        Attachment HDFS-3005.patch [ 12517881 ]
        Vinayakumar B made changes -
        Attachment HDFS-3005.patch [ 12517881 ]
        Vinayakumar B made changes -
        Attachment HDFS-3005.patch [ 12516075 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12516075/HDFS-3005.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hdfs.server.common.TestDistributedUpgrade

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1912//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1912//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12516075/HDFS-3005.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.server.common.TestDistributedUpgrade +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1912//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1912//console This message is automatically generated.
        Vinayakumar B made changes -
        Attachment HDFS-3005.patch [ 12516075 ]
        Vinayakumar B made changes -
        Field Original Value New Value
        Status Open [ 1 ] Patch Available [ 10002 ]
        Affects Version/s 0.24.0 [ 12317653 ]
        Target Version/s 0.24.0 [ 12317653 ]
        Hide
        Vinayakumar B added a comment -

        Just changed HashMap to ConcurrentHashMap. So Not adding any Tests. Tested Concurrent Operations on ConcurrentHashMap with Sample Program.

        Show
        Vinayakumar B added a comment - Just changed HashMap to ConcurrentHashMap. So Not adding any Tests. Tested Concurrent Operations on ConcurrentHashMap with Sample Program.
        Hide
        Suresh Srinivas added a comment -

        I added that comment when doing Federation work. The problem was with the existing code. I wanted to cleanup that part of the code and did not get to it. Now that it is an issue, lets fix it

        Show
        Suresh Srinivas added a comment - I added that comment when doing Federation work. The problem was with the existing code. I wanted to cleanup that part of the code and did not get to it. Now that it is an issue, lets fix it
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Interesting, this was already identified as a potential bug earlier. We have a comment saying,

              // TODO valid synchronization
        
        Show
        Tsz Wo Nicholas Sze added a comment - Interesting, this was already identified as a potential bug earlier. We have a comment saying, // TODO valid synchronization
        Tsz Wo Nicholas Sze created issue -

          People

          • Assignee:
            Tsz Wo Nicholas Sze
            Reporter:
            Tsz Wo Nicholas Sze
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development