Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10319

Balancer should not try to pair storages with different types

    Details

    • Hadoop Flags:
      Reviewed

      Description

      This is a performance bug – Balancer may pair a source datanode and a target datanode with different storage types. Fortunately, it will fail schedule any blocks in such pair since it will find out that the storage types are not matched later on.

      The bug won't lead to incorrect results.

      1. h10319_20160420.patch
        1 kB
        Tsz Wo Nicholas Sze

        Activity

        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        h10319_20160420.patch: matches storage types.

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - h10319_20160420.patch: matches storage types.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 11s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 36s trunk passed
        +1 compile 0m 42s trunk passed with JDK v1.8.0_77
        +1 compile 0m 41s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 21s trunk passed
        +1 mvnsite 0m 50s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 findbugs 1m 57s trunk passed
        +1 javadoc 1m 7s trunk passed with JDK v1.8.0_77
        +1 javadoc 1m 44s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 46s the patch passed
        +1 compile 0m 38s the patch passed with JDK v1.8.0_77
        +1 javac 0m 38s the patch passed
        +1 compile 0m 43s the patch passed with JDK v1.7.0_95
        +1 javac 0m 43s the patch passed
        +1 checkstyle 0m 21s the patch passed
        +1 mvnsite 0m 54s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 21s the patch passed
        +1 javadoc 1m 11s the patch passed with JDK v1.8.0_77
        +1 javadoc 1m 47s the patch passed with JDK v1.7.0_95
        -1 unit 64m 0s hadoop-hdfs in the patch failed with JDK v1.8.0_77.
        -1 unit 54m 6s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 20s Patch does not generate ASF License warnings.
        143m 55s



        Reason Tests
        JDK v1.8.0_77 Failed junit tests hadoop.hdfs.TestFileAppend
          hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality
          hadoop.hdfs.TestHFlush
        JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestHFlush



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799810/h10319_20160420.patch
        JIRA Issue HDFS-10319
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux b345e4e9e476 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ad36fa6
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15223/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15223/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15223/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15223/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15223/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15223/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 36s trunk passed +1 compile 0m 42s trunk passed with JDK v1.8.0_77 +1 compile 0m 41s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 50s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 57s trunk passed +1 javadoc 1m 7s trunk passed with JDK v1.8.0_77 +1 javadoc 1m 44s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 46s the patch passed +1 compile 0m 38s the patch passed with JDK v1.8.0_77 +1 javac 0m 38s the patch passed +1 compile 0m 43s the patch passed with JDK v1.7.0_95 +1 javac 0m 43s the patch passed +1 checkstyle 0m 21s the patch passed +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 21s the patch passed +1 javadoc 1m 11s the patch passed with JDK v1.8.0_77 +1 javadoc 1m 47s the patch passed with JDK v1.7.0_95 -1 unit 64m 0s hadoop-hdfs in the patch failed with JDK v1.8.0_77. -1 unit 54m 6s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 20s Patch does not generate ASF License warnings. 143m 55s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.hdfs.TestFileAppend   hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality   hadoop.hdfs.TestHFlush JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestHFlush Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799810/h10319_20160420.patch JIRA Issue HDFS-10319 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b345e4e9e476 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ad36fa6 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15223/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15223/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15223/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15223/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15223/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15223/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        cnauroth Chris Nauroth added a comment -

        +1 for the patch. The test failures look unrelated. Thank you, Tsz Wo Nicholas Sze.

        Show
        cnauroth Chris Nauroth added a comment - +1 for the patch. The test failures look unrelated. Thank you, Tsz Wo Nicholas Sze .
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        Thanks Chris for reviewing the patch.

        I have committed this.

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - Thanks Chris for reviewing the patch. I have committed this.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9650 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9650/)
        HDFS-10319. Balancer should not try to pair storages with different (szetszwo: rev bbce1d525e0016c9b8e573b86af3c87aa39582bd)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9650 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9650/ ) HDFS-10319 . Balancer should not try to pair storages with different (szetszwo: rev bbce1d525e0016c9b8e573b86af3c87aa39582bd) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Closing the JIRA as part of 2.7.3 release.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.

          People

          • Assignee:
            szetszwo Tsz Wo Nicholas Sze
            Reporter:
            szetszwo Tsz Wo Nicholas Sze
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development