Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: erasure-coding, namenode
    • Labels:
      None

      Description

      For an EC block, it is possible we have enough internal blocks but without enough racks. The current reconstruction code calls BlockManager#isInNewRack to check if the target node can increase the total rack number for the case, which compares the target node's rack with source node racks:

          for (DatanodeDescriptor src : srcs) {
            if (src.getNetworkLocation().equals(target.getNetworkLocation())) {
              return false;
            }
          }
      

      However here the srcs may include a decommissioning node, in which case we should allow the target node to be in the same rack with it.

      For e.g., suppose we have 11 nodes: h1 ~ h11, which are located in racks r1, r1, r2, r2, r3, r3, r4, r4, r5, r5, r6, respectively. In case that an EC block has 9 live internal blocks on (h1~h8 + h11), and one internal block on h9 which is to be decommissioned. The current code will not choose h10 for reconstruction because isInNewRack thinks h10 is on the same rack with h9.

        Activity

        Hide
        jingzhao Jing Zhao added a comment -

        Upload a patch to fix the issue. The patch also includes a unit test that reproduces the example in the description.

        Show
        jingzhao Jing Zhao added a comment - Upload a patch to fix the issue. The patch also includes a unit test that reproduces the example in the description.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 54s trunk passed
        +1 compile 0m 50s trunk passed
        +1 checkstyle 0m 30s trunk passed
        +1 mvnsite 1m 0s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 findbugs 1m 55s trunk passed
        +1 javadoc 1m 3s trunk passed
        +1 mvninstall 1m 3s the patch passed
        +1 compile 0m 54s the patch passed
        +1 javac 0m 54s the patch passed
        -0 checkstyle 0m 29s hadoop-hdfs-project/hadoop-hdfs: The patch generated 10 new + 130 unchanged - 4 fixed = 140 total (was 134)
        +1 mvnsite 1m 1s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 0s the patch passed
        +1 javadoc 1m 3s the patch passed
        +1 unit 81m 26s hadoop-hdfs in the patch passed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        103m 47s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-10968
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831857/HDFS-10968.000.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 97b67b60f168 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / c5ca216
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17035/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17035/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17035/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 54s trunk passed +1 compile 0m 50s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 1m 0s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 55s trunk passed +1 javadoc 1m 3s trunk passed +1 mvninstall 1m 3s the patch passed +1 compile 0m 54s the patch passed +1 javac 0m 54s the patch passed -0 checkstyle 0m 29s hadoop-hdfs-project/hadoop-hdfs: The patch generated 10 new + 130 unchanged - 4 fixed = 140 total (was 134) +1 mvnsite 1m 1s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 0s the patch passed +1 javadoc 1m 3s the patch passed +1 unit 81m 26s hadoop-hdfs in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 103m 47s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10968 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831857/HDFS-10968.000.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 97b67b60f168 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c5ca216 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17035/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17035/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17035/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        Good catch!

        +1 the patch looks good.

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - Good catch! +1 the patch looks good.
        Hide
        jingzhao Jing Zhao added a comment -

        Thanks for the review, Nicholas! I've committed this into trunk.

        Show
        jingzhao Jing Zhao added a comment - Thanks for the review, Nicholas! I've committed this into trunk.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10573 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10573/)
        HDFS-10968. BlockManager#isInNewRack should consider decommissioning (jing9: rev 4d106213c0f4835b723c9a50bd8080a9017122d7)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReconstructStripedBlocksWithRackAwareness.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10573 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10573/ ) HDFS-10968 . BlockManager#isInNewRack should consider decommissioning (jing9: rev 4d106213c0f4835b723c9a50bd8080a9017122d7) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReconstructStripedBlocksWithRackAwareness.java

          People

          • Assignee:
            jingzhao Jing Zhao
            Reporter:
            jingzhao Jing Zhao
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development