Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10297

Increase default balance bandwidth and concurrent moves

    Details

    • Target Version/s:

      Description

      Adjust the default values to better support the current level of customer host and network configurations.

      Increase the default for property dfs.datanode.balance.bandwidthPerSec from 1 to 10 MB. Apply to DN. 10 MB/s is about 10% of the GbE network.

      Increase the default for property dfs.datanode.balance.max.concurrent.moves from 5 to 50. Apply to DN and Balancer. The default number of DN receiver threads is 4096. The default number of balancer mover threads is 1000.

      1. HDFS-10297.001.patch
        3 kB
        John Zhuge
      2. HDFS-10297.002.patch
        3 kB
        John Zhuge
      3. HDFS-10297.003.branch-2.8.patch
        3 kB
        John Zhuge
      4. HDFS-10297.003.patch
        3 kB
        John Zhuge

        Activity

        Hide
        jzhuge John Zhuge added a comment -

        Patch 001:

        • Change values in hdfs-default.xml
        • Change values of DFS_DATANODE_BALANCE_BANDWIDTHPERSEC_DEFAULT and DFS_DATANODE_BALANCE_MAX_NUM_CONCURRENT_MOVES_DEFAULT

        Test output:

        $ ( cd hadoop-hdfs-project/hadoop-hdfs ; mvn test -Dtest=TestBalancer#testBalancer2 )
        $ grep 'concurrent\.moves\|Number threads for balancing' hadoop-hdfs-project/hadoop-hdfs/target/surefire-reports/*output*
        2016-04-15 10:46:58,563 [Thread-0] INFO  datanode.DataNode (DataXceiverServer.java:<init>(79)) - Number threads for balancing is 50
        2016-04-15 10:46:58,781 [Thread-0] INFO  datanode.DataNode (DataXceiverServer.java:<init>(79)) - Number threads for balancing is 50
        2016-04-15 10:46:59,942 [Thread-0] INFO  datanode.DataNode (DataXceiverServer.java:<init>(79)) - Number threads for balancing is 50
        2016-04-15 10:47:00,153 [Thread-0] INFO  balancer.Balancer (Balancer.java:getInt(240)) - dfs.datanode.balance.max.concurrent.moves = 50 (default=50)
        2016-04-15 10:47:04,205 [Thread-0] INFO  balancer.Balancer (Balancer.java:getInt(240)) - dfs.datanode.balance.max.concurrent.moves = 50 (default=50)
        [jzhuge@jzhuge-MBP hadoop](trunk *)$ grep 'bandwidthPer\|Balancing bandwidth' hadoop-hdfs-project/hadoop-hdfs/target/surefire-reports/*output*
        2016-04-15 10:46:58,563 [Thread-0] INFO  datanode.DataNode (DataXceiverServer.java:<init>(78)) - Balancing bandwidth is 10485760 bytes/s
        2016-04-15 10:46:58,781 [Thread-0] INFO  datanode.DataNode (DataXceiverServer.java:<init>(78)) - Balancing bandwidth is 10485760 bytes/s
        2016-04-15 10:46:59,942 [Thread-0] INFO  datanode.DataNode (DataXceiverServer.java:<init>(78)) - Balancing bandwidth is 10485760 bytes/s
        
        Show
        jzhuge John Zhuge added a comment - Patch 001: Change values in hdfs-default.xml Change values of DFS_DATANODE_BALANCE_BANDWIDTHPERSEC_DEFAULT and DFS_DATANODE_BALANCE_MAX_NUM_CONCURRENT_MOVES_DEFAULT Test output: $ ( cd hadoop-hdfs-project/hadoop-hdfs ; mvn test -Dtest=TestBalancer#testBalancer2 ) $ grep 'concurrent\.moves\|Number threads for balancing' hadoop-hdfs-project/hadoop-hdfs/target/surefire-reports/*output* 2016-04-15 10:46:58,563 [Thread-0] INFO datanode.DataNode (DataXceiverServer.java:<init>(79)) - Number threads for balancing is 50 2016-04-15 10:46:58,781 [Thread-0] INFO datanode.DataNode (DataXceiverServer.java:<init>(79)) - Number threads for balancing is 50 2016-04-15 10:46:59,942 [Thread-0] INFO datanode.DataNode (DataXceiverServer.java:<init>(79)) - Number threads for balancing is 50 2016-04-15 10:47:00,153 [Thread-0] INFO balancer.Balancer (Balancer.java:getInt(240)) - dfs.datanode.balance.max.concurrent.moves = 50 (default=50) 2016-04-15 10:47:04,205 [Thread-0] INFO balancer.Balancer (Balancer.java:getInt(240)) - dfs.datanode.balance.max.concurrent.moves = 50 (default=50) [jzhuge@jzhuge-MBP hadoop](trunk *)$ grep 'bandwidthPer\|Balancing bandwidth' hadoop-hdfs-project/hadoop-hdfs/target/surefire-reports/*output* 2016-04-15 10:46:58,563 [Thread-0] INFO datanode.DataNode (DataXceiverServer.java:<init>(78)) - Balancing bandwidth is 10485760 bytes/s 2016-04-15 10:46:58,781 [Thread-0] INFO datanode.DataNode (DataXceiverServer.java:<init>(78)) - Balancing bandwidth is 10485760 bytes/s 2016-04-15 10:46:59,942 [Thread-0] INFO datanode.DataNode (DataXceiverServer.java:<init>(78)) - Balancing bandwidth is 10485760 bytes/s
        Hide
        jingzhao Jing Zhao added a comment -

        John Zhuge, thanks for reporting the issue and providing a patch. I noticed you set the "Fix Version" to 2.8.0. We usually update "Fix Version" only after the patch has been committed. So currently I removed that field for you.

        Show
        jingzhao Jing Zhao added a comment - John Zhuge , thanks for reporting the issue and providing a patch. I noticed you set the "Fix Version" to 2.8.0. We usually update "Fix Version" only after the patch has been committed. So currently I removed that field for you.
        Hide
        jzhuge John Zhuge added a comment -

        Thanks Jing Zhao.

        Show
        jzhuge John Zhuge added a comment - Thanks Jing Zhao .
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 53s trunk passed
        +1 compile 0m 40s trunk passed with JDK v1.8.0_77
        +1 compile 0m 39s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 26s trunk passed
        +1 mvnsite 0m 51s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 2m 0s trunk passed
        +1 javadoc 1m 5s trunk passed with JDK v1.8.0_77
        +1 javadoc 1m 46s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 45s the patch passed
        +1 compile 0m 39s the patch passed with JDK v1.8.0_77
        +1 javac 0m 39s the patch passed
        +1 compile 0m 38s the patch passed with JDK v1.7.0_95
        +1 javac 0m 38s the patch passed
        -1 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: patch generated 2 new + 393 unchanged - 2 fixed = 395 total (was 395)
        +1 mvnsite 0m 49s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 2m 10s the patch passed
        +1 javadoc 1m 2s the patch passed with JDK v1.8.0_77
        +1 javadoc 1m 43s the patch passed with JDK v1.7.0_95
        -1 unit 56m 42s hadoop-hdfs in the patch failed with JDK v1.8.0_77.
        -1 unit 53m 18s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 22s Patch does not generate ASF License warnings.
        135m 38s



        Reason Tests
        JDK v1.8.0_77 Failed junit tests hadoop.hdfs.TestHFlush
          hadoop.hdfs.server.namenode.TestEditLog
          hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
        JDK v1.7.0_95 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798988/HDFS-10297.001.patch
        JIRA Issue HDFS-10297
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 5ffad89c50d7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / fdbafbc
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15171/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15171/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15171/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15171/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15171/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15171/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15171/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 53s trunk passed +1 compile 0m 40s trunk passed with JDK v1.8.0_77 +1 compile 0m 39s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 26s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 2m 0s trunk passed +1 javadoc 1m 5s trunk passed with JDK v1.8.0_77 +1 javadoc 1m 46s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 45s the patch passed +1 compile 0m 39s the patch passed with JDK v1.8.0_77 +1 javac 0m 39s the patch passed +1 compile 0m 38s the patch passed with JDK v1.7.0_95 +1 javac 0m 38s the patch passed -1 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: patch generated 2 new + 393 unchanged - 2 fixed = 395 total (was 395) +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 10s the patch passed +1 javadoc 1m 2s the patch passed with JDK v1.8.0_77 +1 javadoc 1m 43s the patch passed with JDK v1.7.0_95 -1 unit 56m 42s hadoop-hdfs in the patch failed with JDK v1.8.0_77. -1 unit 53m 18s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 135m 38s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.hdfs.TestHFlush   hadoop.hdfs.server.namenode.TestEditLog   hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead JDK v1.7.0_95 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798988/HDFS-10297.001.patch JIRA Issue HDFS-10297 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 5ffad89c50d7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / fdbafbc Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15171/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15171/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15171/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15171/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15171/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15171/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15171/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        jzhuge John Zhuge added a comment -

        Patch 002:

        • Fix checkstyle errors
        Show
        jzhuge John Zhuge added a comment - Patch 002: Fix checkstyle errors
        Hide
        jzhuge John Zhuge added a comment -

        Unit test failures look unrelated.

        Show
        jzhuge John Zhuge added a comment - Unit test failures look unrelated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 24s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 10m 18s trunk passed
        +1 compile 1m 48s trunk passed with JDK v1.8.0_77
        +1 compile 1m 18s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 44s trunk passed
        +1 mvnsite 1m 12s trunk passed
        +1 mvneclipse 0m 17s trunk passed
        +1 findbugs 2m 39s trunk passed
        +1 javadoc 2m 20s trunk passed with JDK v1.8.0_77
        +1 javadoc 3m 21s trunk passed with JDK v1.7.0_95
        +1 mvninstall 1m 24s the patch passed
        +1 compile 1m 46s the patch passed with JDK v1.8.0_77
        +1 javac 1m 46s the patch passed
        +1 compile 1m 17s the patch passed with JDK v1.7.0_95
        +1 javac 1m 17s the patch passed
        -1 checkstyle 0m 39s hadoop-hdfs-project/hadoop-hdfs: patch generated 1 new + 392 unchanged - 3 fixed = 393 total (was 395)
        +1 mvnsite 1m 11s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 2m 24s the patch passed
        +1 javadoc 1m 19s the patch passed with JDK v1.8.0_77
        +1 javadoc 1m 54s the patch passed with JDK v1.7.0_95
        -1 unit 110m 57s hadoop-hdfs in the patch failed with JDK v1.8.0_77.
        -1 unit 84m 45s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 22s Patch does not generate ASF License warnings.
        235m 15s



        Reason Tests
        JDK v1.8.0_77 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeUUID
          hadoop.hdfs.TestDFSUpgradeFromImage
          hadoop.hdfs.server.namenode.ha.TestEditLogTailer
          hadoop.hdfs.TestPersistBlocks
          hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
          hadoop.hdfs.security.TestDelegationTokenForProxyUser
          hadoop.hdfs.server.datanode.TestDataNodeLifeline
          hadoop.hdfs.TestLocalDFS
          hadoop.hdfs.TestFileAppend
          hadoop.hdfs.server.namenode.ha.TestHAAppend
          hadoop.hdfs.TestEncryptionZones
          hadoop.hdfs.server.datanode.TestDirectoryScanner
        JDK v1.8.0_77 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement
        JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeUUID
          hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
          hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs
          hadoop.hdfs.server.namenode.TestDecommissioningStatus
          hadoop.hdfs.qjournal.TestSecureNNWithQJM
          hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
          hadoop.hdfs.server.datanode.TestDirectoryScanner



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799082/HDFS-10297.002.patch
        JIRA Issue HDFS-10297
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux da544105d276 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / cab9cba
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15178/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15178/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15178/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15178/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15178/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15178/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15178/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 24s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 10m 18s trunk passed +1 compile 1m 48s trunk passed with JDK v1.8.0_77 +1 compile 1m 18s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 44s trunk passed +1 mvnsite 1m 12s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 2m 39s trunk passed +1 javadoc 2m 20s trunk passed with JDK v1.8.0_77 +1 javadoc 3m 21s trunk passed with JDK v1.7.0_95 +1 mvninstall 1m 24s the patch passed +1 compile 1m 46s the patch passed with JDK v1.8.0_77 +1 javac 1m 46s the patch passed +1 compile 1m 17s the patch passed with JDK v1.7.0_95 +1 javac 1m 17s the patch passed -1 checkstyle 0m 39s hadoop-hdfs-project/hadoop-hdfs: patch generated 1 new + 392 unchanged - 3 fixed = 393 total (was 395) +1 mvnsite 1m 11s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 24s the patch passed +1 javadoc 1m 19s the patch passed with JDK v1.8.0_77 +1 javadoc 1m 54s the patch passed with JDK v1.7.0_95 -1 unit 110m 57s hadoop-hdfs in the patch failed with JDK v1.8.0_77. -1 unit 84m 45s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 235m 15s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeUUID   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.server.datanode.TestDataNodeLifeline   hadoop.hdfs.TestLocalDFS   hadoop.hdfs.TestFileAppend   hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.server.datanode.TestDirectoryScanner JDK v1.8.0_77 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeUUID   hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot   hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799082/HDFS-10297.002.patch JIRA Issue HDFS-10297 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux da544105d276 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / cab9cba Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15178/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15178/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15178/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15178/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15178/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15178/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15178/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        jzhuge John Zhuge added a comment -

        Patch 003:

        • Fix checkstyle
        Show
        jzhuge John Zhuge added a comment - Patch 003: Fix checkstyle
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 9s trunk passed
        +1 compile 0m 52s trunk passed with JDK v1.8.0_77
        +1 compile 0m 44s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 24s trunk passed
        +1 mvnsite 0m 53s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 2m 0s trunk passed
        +1 javadoc 1m 14s trunk passed with JDK v1.8.0_77
        +1 javadoc 1m 55s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 48s the patch passed
        +1 compile 0m 49s the patch passed with JDK v1.8.0_77
        +1 javac 0m 49s the patch passed
        +1 compile 0m 43s the patch passed with JDK v1.7.0_95
        +1 javac 0m 43s the patch passed
        +1 checkstyle 0m 22s hadoop-hdfs-project/hadoop-hdfs: patch generated 0 new + 392 unchanged - 3 fixed = 392 total (was 395)
        +1 mvnsite 0m 51s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 0s The patch has no ill-formed XML file.
        +1 findbugs 2m 9s the patch passed
        +1 javadoc 1m 14s the patch passed with JDK v1.8.0_77
        +1 javadoc 1m 50s the patch passed with JDK v1.7.0_95
        -1 unit 90m 51s hadoop-hdfs in the patch failed with JDK v1.8.0_77.
        -1 unit 78m 59s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 20s Patch does not generate ASF License warnings.
        197m 1s



        Reason Tests
        JDK v1.8.0_77 Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer
          hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
          hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
          hadoop.hdfs.TestFileAppend
          hadoop.hdfs.server.datanode.TestDataNodeMXBean
          hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
          hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs
        JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage
          hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
          hadoop.hdfs.server.namenode.TestReconstructStripedBlocks
          hadoop.hdfs.server.namenode.TestEditLog
          hadoop.hdfs.server.datanode.TestDirectoryScanner



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799112/HDFS-10297.003.patch
        JIRA Issue HDFS-10297
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 24b71fe7d361 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / cab9cba
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15180/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15180/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15180/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15180/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15180/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15180/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 9s trunk passed +1 compile 0m 52s trunk passed with JDK v1.8.0_77 +1 compile 0m 44s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 24s trunk passed +1 mvnsite 0m 53s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 2m 0s trunk passed +1 javadoc 1m 14s trunk passed with JDK v1.8.0_77 +1 javadoc 1m 55s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 48s the patch passed +1 compile 0m 49s the patch passed with JDK v1.8.0_77 +1 javac 0m 49s the patch passed +1 compile 0m 43s the patch passed with JDK v1.7.0_95 +1 javac 0m 43s the patch passed +1 checkstyle 0m 22s hadoop-hdfs-project/hadoop-hdfs: patch generated 0 new + 392 unchanged - 3 fixed = 392 total (was 395) +1 mvnsite 0m 51s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 2m 9s the patch passed +1 javadoc 1m 14s the patch passed with JDK v1.8.0_77 +1 javadoc 1m 50s the patch passed with JDK v1.7.0_95 -1 unit 90m 51s hadoop-hdfs in the patch failed with JDK v1.8.0_77. -1 unit 78m 59s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 20s Patch does not generate ASF License warnings. 197m 1s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl   hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.TestFileAppend   hadoop.hdfs.server.datanode.TestDataNodeMXBean   hadoop.hdfs.server.blockmanagement.TestReplicationPolicy   hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.server.namenode.TestReconstructStripedBlocks   hadoop.hdfs.server.namenode.TestEditLog   hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799112/HDFS-10297.003.patch JIRA Issue HDFS-10297 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 24b71fe7d361 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / cab9cba Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15180/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15180/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15180/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15180/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15180/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15180/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        jzhuge John Zhuge added a comment -

        Could someone kindly review this simple patch?

        Show
        jzhuge John Zhuge added a comment - Could someone kindly review this simple patch?
        Hide
        andrew.wang Andrew Wang added a comment -

        LGTM +1, will commit shortly.

        Show
        andrew.wang Andrew Wang added a comment - LGTM +1, will commit shortly.
        Hide
        andrew.wang Andrew Wang added a comment -

        Committed to trunk and branch-2, branch-2.8 wasn't clean. John, if you want it in 2.8, do you mind providing a branch-2.8 patch? Else we can just resolve.

        Show
        andrew.wang Andrew Wang added a comment - Committed to trunk and branch-2, branch-2.8 wasn't clean. John, if you want it in 2.8, do you mind providing a branch-2.8 patch? Else we can just resolve.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9672 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9672/)
        HDFS-10297. Increase default balance bandwidth and concurrent moves. (wang: rev 6be22ddbf1b6f3c19ec70c63ddb8f5519d18dd72)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9672 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9672/ ) HDFS-10297 . Increase default balance bandwidth and concurrent moves. (wang: rev 6be22ddbf1b6f3c19ec70c63ddb8f5519d18dd72) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        Hide
        jzhuge John Zhuge added a comment -

        Thank you Andrew Wang. I will create a patch for 2.8.

        Show
        jzhuge John Zhuge added a comment - Thank you Andrew Wang . I will create a patch for 2.8.
        Hide
        jzhuge John Zhuge added a comment -

        Wow, so many properties of file hdfs-default.xml in branch-2 (line 2972 to 3984) are NOT in branch-2.8 or 2.7 including dfs.datanode.balance.max.concurrent.moves?!

        Show
        jzhuge John Zhuge added a comment - Wow, so many properties of file hdfs-default.xml in branch-2 (line 2972 to 3984) are NOT in branch-2.8 or 2.7 including dfs.datanode.balance.max.concurrent.moves ?!
        Hide
        jzhuge John Zhuge added a comment -

        Ray Chiang's HDFS-8356 added the missing properties. Is it applicable to 2.8 and 2.7? My guess is no.

        Show
        jzhuge John Zhuge added a comment - Ray Chiang 's HDFS-8356 added the missing properties. Is it applicable to 2.8 and 2.7? My guess is no.
        Hide
        rchiang Ray Chiang added a comment -

        Yeah, I only did trunk and branch-2 versions for HDFS-8356. I didn't go backwards compatible with all the branch-2 minor versions.

        Show
        rchiang Ray Chiang added a comment - Yeah, I only did trunk and branch-2 versions for HDFS-8356 . I didn't go backwards compatible with all the branch-2 minor versions.
        Hide
        jzhuge John Zhuge added a comment -

        Thanks Ray Chiang.

        Show
        jzhuge John Zhuge added a comment - Thanks Ray Chiang .
        Hide
        jzhuge John Zhuge added a comment -

        Upload HDFS-10297.003.branch-2.8.patch for branch 2.8.

        Show
        jzhuge John Zhuge added a comment - Upload HDFS-10297 .003.branch-2.8.patch for branch 2.8.
        Hide
        jzhuge John Zhuge added a comment -

        Andrew Wang, I uploaded a patch for branch-2.8, please merge.

        Show
        jzhuge John Zhuge added a comment - Andrew Wang , I uploaded a patch for branch-2.8, please merge.
        Hide
        andrew.wang Andrew Wang added a comment -

        Pushed, thanks John!

        Show
        andrew.wang Andrew Wang added a comment - Pushed, thanks John!

          People

          • Assignee:
            jzhuge John Zhuge
            Reporter:
            jzhuge John Zhuge
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development