Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10297

Increase default balance bandwidth and concurrent moves

        Details

        • Target Version/s:

          Description

          Adjust the default values to better support the current level of customer host and network configurations.

          Increase the default for property dfs.datanode.balance.bandwidthPerSec from 1 to 10 MB. Apply to DN. 10 MB/s is about 10% of the GbE network.

          Increase the default for property dfs.datanode.balance.max.concurrent.moves from 5 to 50. Apply to DN and Balancer. The default number of DN receiver threads is 4096. The default number of balancer mover threads is 1000.

          1. Text File
            HDFS-10297.001.patch
            3 kB
            John Zhuge
          2. Text File
            HDFS-10297.002.patch
            3 kB
            John Zhuge
          3. Text File
            HDFS-10297.003.branch-2.8.patch
            3 kB
            John Zhuge
          4. Text File
            HDFS-10297.003.patch
            3 kB
            John Zhuge

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            jzhuge John Zhuge added a comment -

            Patch 001:

            • Change values in hdfs-default.xml
            • Change values of DFS_DATANODE_BALANCE_BANDWIDTHPERSEC_DEFAULT and DFS_DATANODE_BALANCE_MAX_NUM_CONCURRENT_MOVES_DEFAULT

            Test output:

            $ ( cd hadoop-hdfs-project/hadoop-hdfs ; mvn test -Dtest=TestBalancer#testBalancer2 )
$ grep 'concurrent\.moves\|Number threads for balancing' hadoop-hdfs-project/hadoop-hdfs/target/surefire-reports/*output*
2016-04-15 10:46:58,563 [Thread-0] INFO  datanode.DataNode (DataXceiverServer.java:<init>(79)) - Number threads for balancing is 50
2016-04-15 10:46:58,781 [Thread-0] INFO  datanode.DataNode (DataXceiverServer.java:<init>(79)) - Number threads for balancing is 50
2016-04-15 10:46:59,942 [Thread-0] INFO  datanode.DataNode (DataXceiverServer.java:<init>(79)) - Number threads for balancing is 50
2016-04-15 10:47:00,153 [Thread-0] INFO  balancer.Balancer (Balancer.java:getInt(240)) - dfs.datanode.balance.max.concurrent.moves = 50 (default=50)
2016-04-15 10:47:04,205 [Thread-0] INFO  balancer.Balancer (Balancer.java:getInt(240)) - dfs.datanode.balance.max.concurrent.moves = 50 (default=50)
[jzhuge@jzhuge-MBP hadoop](trunk *)$ grep 'bandwidthPer\|Balancing bandwidth' hadoop-hdfs-project/hadoop-hdfs/target/surefire-reports/*output*
2016-04-15 10:46:58,563 [Thread-0] INFO  datanode.DataNode (DataXceiverServer.java:<init>(78)) - Balancing bandwidth is 10485760 bytes/s
2016-04-15 10:46:58,781 [Thread-0] INFO  datanode.DataNode (DataXceiverServer.java:<init>(78)) - Balancing bandwidth is 10485760 bytes/s
2016-04-15 10:46:59,942 [Thread-0] INFO  datanode.DataNode (DataXceiverServer.java:<init>(78)) - Balancing bandwidth is 10485760 bytes/s
            Show
            jzhuge John Zhuge added a comment - Patch 001: Change values in hdfs-default.xml Change values of DFS_DATANODE_BALANCE_BANDWIDTHPERSEC_DEFAULT and DFS_DATANODE_BALANCE_MAX_NUM_CONCURRENT_MOVES_DEFAULT Test output: $ ( cd hadoop-hdfs-project/hadoop-hdfs ; mvn test -Dtest=TestBalancer#testBalancer2 ) $ grep 'concurrent\.moves\|Number threads for balancing' hadoop-hdfs-project/hadoop-hdfs/target/surefire-reports/*output* 2016-04-15 10:46:58,563 [Thread-0] INFO datanode.DataNode (DataXceiverServer.java:<init>(79)) - Number threads for balancing is 50 2016-04-15 10:46:58,781 [Thread-0] INFO datanode.DataNode (DataXceiverServer.java:<init>(79)) - Number threads for balancing is 50 2016-04-15 10:46:59,942 [Thread-0] INFO datanode.DataNode (DataXceiverServer.java:<init>(79)) - Number threads for balancing is 50 2016-04-15 10:47:00,153 [Thread-0] INFO balancer.Balancer (Balancer.java:getInt(240)) - dfs.datanode.balance.max.concurrent.moves = 50 (default=50) 2016-04-15 10:47:04,205 [Thread-0] INFO balancer.Balancer (Balancer.java:getInt(240)) - dfs.datanode.balance.max.concurrent.moves = 50 (default=50) [jzhuge@jzhuge-MBP hadoop](trunk *)$ grep 'bandwidthPer\|Balancing bandwidth' hadoop-hdfs-project/hadoop-hdfs/target/surefire-reports/*output* 2016-04-15 10:46:58,563 [Thread-0] INFO datanode.DataNode (DataXceiverServer.java:<init>(78)) - Balancing bandwidth is 10485760 bytes/s 2016-04-15 10:46:58,781 [Thread-0] INFO datanode.DataNode (DataXceiverServer.java:<init>(78)) - Balancing bandwidth is 10485760 bytes/s 2016-04-15 10:46:59,942 [Thread-0] INFO datanode.DataNode (DataXceiverServer.java:<init>(78)) - Balancing bandwidth is 10485760 bytes/s
            Hide
            Permalink
            jingzhao Jing Zhao added a comment -

            John Zhuge, thanks for reporting the issue and providing a patch. I noticed you set the "Fix Version" to 2.8.0. We usually update "Fix Version" only after the patch has been committed. So currently I removed that field for you.

            Show
            jingzhao Jing Zhao added a comment - John Zhuge , thanks for reporting the issue and providing a patch. I noticed you set the "Fix Version" to 2.8.0. We usually update "Fix Version" only after the patch has been committed. So currently I removed that field for you.
            Hide
            Permalink
            jzhuge John Zhuge added a comment -

            Thanks Jing Zhao.

            Show
            jzhuge John Zhuge added a comment - Thanks Jing Zhao .
            Hide
            Permalink
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 18s Docker mode activated.
            +1 @author 0m 0s The patch does not contain any @author tags.
            -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
            +1 mvninstall 6m 53s trunk passed
            +1 compile 0m 40s trunk passed with JDK v1.8.0_77
            +1 compile 0m 39s trunk passed with JDK v1.7.0_95
            +1 checkstyle 0m 26s trunk passed
            +1 mvnsite 0m 51s trunk passed
            +1 mvneclipse 0m 12s trunk passed
            +1 findbugs 2m 0s trunk passed
            +1 javadoc 1m 5s trunk passed with JDK v1.8.0_77
            +1 javadoc 1m 46s trunk passed with JDK v1.7.0_95
            +1 mvninstall 0m 45s the patch passed
            +1 compile 0m 39s the patch passed with JDK v1.8.0_77
            +1 javac 0m 39s the patch passed
            +1 compile 0m 38s the patch passed with JDK v1.7.0_95
            +1 javac 0m 38s the patch passed
            -1 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: patch generated 2 new + 393 unchanged - 2 fixed = 395 total (was 395)
            +1 mvnsite 0m 49s the patch passed
            +1 mvneclipse 0m 11s the patch passed
            +1 whitespace 0m 0s Patch has no whitespace issues.
            +1 xml 0m 1s The patch has no ill-formed XML file.
            +1 findbugs 2m 10s the patch passed
            +1 javadoc 1m 2s the patch passed with JDK v1.8.0_77
            +1 javadoc 1m 43s the patch passed with JDK v1.7.0_95
            -1 unit 56m 42s hadoop-hdfs in the patch failed with JDK v1.8.0_77.
            -1 unit 53m 18s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
            +1 asflicense 0m 22s Patch does not generate ASF License warnings.
            135m 38s



            Reason Tests
            JDK v1.8.0_77 Failed junit tests hadoop.hdfs.TestHFlush
              hadoop.hdfs.server.namenode.TestEditLog
              hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
            JDK v1.7.0_95 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead



            Subsystem Report/Notes
            Docker Image:yetus/hadoop:fbe3e86
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798988/HDFS-10297.001.patch
            JIRA Issue HDFS-10297
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
            uname Linux 5ffad89c50d7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
            git revision trunk / fdbafbc
            Default Java 1.7.0_95
            Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
            findbugs v3.0.0
            checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15171/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
            unit https://builds.apache.org/job/PreCommit-HDFS-Build/15171/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt
            unit https://builds.apache.org/job/PreCommit-HDFS-Build/15171/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
            unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15171/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15171/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
            JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15171/testReport/
            modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
            Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15171/console
            Powered by Apache Yetus 0.2.0 http://yetus.apache.org

            This message was automatically generated.

            Show
            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 53s trunk passed +1 compile 0m 40s trunk passed with JDK v1.8.0_77 +1 compile 0m 39s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 26s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 2m 0s trunk passed +1 javadoc 1m 5s trunk passed with JDK v1.8.0_77 +1 javadoc 1m 46s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 45s the patch passed +1 compile 0m 39s the patch passed with JDK v1.8.0_77 +1 javac 0m 39s the patch passed +1 compile 0m 38s the patch passed with JDK v1.7.0_95 +1 javac 0m 38s the patch passed -1 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: patch generated 2 new + 393 unchanged - 2 fixed = 395 total (was 395) +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 10s the patch passed +1 javadoc 1m 2s the patch passed with JDK v1.8.0_77 +1 javadoc 1m 43s the patch passed with JDK v1.7.0_95 -1 unit 56m 42s hadoop-hdfs in the patch failed with JDK v1.8.0_77. -1 unit 53m 18s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 135m 38s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.hdfs.TestHFlush   hadoop.hdfs.server.namenode.TestEditLog   hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead JDK v1.7.0_95 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798988/HDFS-10297.001.patch JIRA Issue HDFS-10297 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 5ffad89c50d7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / fdbafbc Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15171/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15171/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15171/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15171/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15171/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15171/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15171/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
            Hide
            Permalink
            jzhuge John Zhuge added a comment -

            Patch 002:

            • Fix checkstyle errors
            Show
            jzhuge John Zhuge added a comment - Patch 002: Fix checkstyle errors
            Hide
            Permalink
            jzhuge John Zhuge added a comment -

            Unit test failures look unrelated.

            Show
            jzhuge John Zhuge added a comment - Unit test failures look unrelated.
            Hide
            Permalink
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 24s Docker mode activated.
            +1 @author 0m 0s The patch does not contain any @author tags.
            -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
            +1 mvninstall 10m 18s trunk passed
            +1 compile 1m 48s trunk passed with JDK v1.8.0_77
            +1 compile 1m 18s trunk passed with JDK v1.7.0_95
            +1 checkstyle 0m 44s trunk passed
            +1 mvnsite 1m 12s trunk passed
            +1 mvneclipse 0m 17s trunk passed
            +1 findbugs 2m 39s trunk passed
            +1 javadoc 2m 20s trunk passed with JDK v1.8.0_77
            +1 javadoc 3m 21s trunk passed with JDK v1.7.0_95
            +1 mvninstall 1m 24s the patch passed
            +1 compile 1m 46s the patch passed with JDK v1.8.0_77
            +1 javac 1m 46s the patch passed
            +1 compile 1m 17s the patch passed with JDK v1.7.0_95
            +1 javac 1m 17s the patch passed
            -1 checkstyle 0m 39s hadoop-hdfs-project/hadoop-hdfs: patch generated 1 new + 392 unchanged - 3 fixed = 393 total (was 395)
            +1 mvnsite 1m 11s the patch passed
            +1 mvneclipse 0m 13s the patch passed
            +1 whitespace 0m 0s Patch has no whitespace issues.
            +1 xml 0m 1s The patch has no ill-formed XML file.
            +1 findbugs 2m 24s the patch passed
            +1 javadoc 1m 19s the patch passed with JDK v1.8.0_77
            +1 javadoc 1m 54s the patch passed with JDK v1.7.0_95
            -1 unit 110m 57s hadoop-hdfs in the patch failed with JDK v1.8.0_77.
            -1 unit 84m 45s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
            +1 asflicense 0m 22s Patch does not generate ASF License warnings.
            235m 15s



            Reason Tests
            JDK v1.8.0_77 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeUUID
              hadoop.hdfs.TestDFSUpgradeFromImage
              hadoop.hdfs.server.namenode.ha.TestEditLogTailer
              hadoop.hdfs.TestPersistBlocks
              hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
              hadoop.hdfs.security.TestDelegationTokenForProxyUser
              hadoop.hdfs.server.datanode.TestDataNodeLifeline
              hadoop.hdfs.TestLocalDFS
              hadoop.hdfs.TestFileAppend
              hadoop.hdfs.server.namenode.ha.TestHAAppend
              hadoop.hdfs.TestEncryptionZones
              hadoop.hdfs.server.datanode.TestDirectoryScanner
            JDK v1.8.0_77 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement
            JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeUUID
              hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
              hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs
              hadoop.hdfs.server.namenode.TestDecommissioningStatus
              hadoop.hdfs.qjournal.TestSecureNNWithQJM
              hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
              hadoop.hdfs.server.datanode.TestDirectoryScanner



            Subsystem Report/Notes
            Docker Image:yetus/hadoop:fbe3e86
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799082/HDFS-10297.002.patch
            JIRA Issue HDFS-10297
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
            uname Linux da544105d276 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
            git revision trunk / cab9cba
            Default Java 1.7.0_95
            Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
            findbugs v3.0.0
            checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15178/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
            unit https://builds.apache.org/job/PreCommit-HDFS-Build/15178/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt
            unit https://builds.apache.org/job/PreCommit-HDFS-Build/15178/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
            unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15178/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15178/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
            JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15178/testReport/
            modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
            Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15178/console
            Powered by Apache Yetus 0.2.0 http://yetus.apache.org

            This message was automatically generated.

            Show
            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 24s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 10m 18s trunk passed +1 compile 1m 48s trunk passed with JDK v1.8.0_77 +1 compile 1m 18s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 44s trunk passed +1 mvnsite 1m 12s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 2m 39s trunk passed +1 javadoc 2m 20s trunk passed with JDK v1.8.0_77 +1 javadoc 3m 21s trunk passed with JDK v1.7.0_95 +1 mvninstall 1m 24s the patch passed +1 compile 1m 46s the patch passed with JDK v1.8.0_77 +1 javac 1m 46s the patch passed +1 compile 1m 17s the patch passed with JDK v1.7.0_95 +1 javac 1m 17s the patch passed -1 checkstyle 0m 39s hadoop-hdfs-project/hadoop-hdfs: patch generated 1 new + 392 unchanged - 3 fixed = 393 total (was 395) +1 mvnsite 1m 11s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 24s the patch passed +1 javadoc 1m 19s the patch passed with JDK v1.8.0_77 +1 javadoc 1m 54s the patch passed with JDK v1.7.0_95 -1 unit 110m 57s hadoop-hdfs in the patch failed with JDK v1.8.0_77. -1 unit 84m 45s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 235m 15s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeUUID   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.server.datanode.TestDataNodeLifeline   hadoop.hdfs.TestLocalDFS   hadoop.hdfs.TestFileAppend   hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.server.datanode.TestDirectoryScanner JDK v1.8.0_77 Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeUUID   hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot   hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799082/HDFS-10297.002.patch JIRA Issue HDFS-10297 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux da544105d276 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / cab9cba Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15178/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15178/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15178/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15178/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15178/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15178/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15178/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
            Hide
            Permalink
            jzhuge John Zhuge added a comment -

            Patch 003:

            • Fix checkstyle
            Show
            jzhuge John Zhuge added a comment - Patch 003: Fix checkstyle
            Hide
            Permalink
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 15s Docker mode activated.
            +1 @author 0m 0s The patch does not contain any @author tags.
            -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
            +1 mvninstall 7m 9s trunk passed
            +1 compile 0m 52s trunk passed with JDK v1.8.0_77
            +1 compile 0m 44s trunk passed with JDK v1.7.0_95
            +1 checkstyle 0m 24s trunk passed
            +1 mvnsite 0m 53s trunk passed
            +1 mvneclipse 0m 13s trunk passed
            +1 findbugs 2m 0s trunk passed
            +1 javadoc 1m 14s trunk passed with JDK v1.8.0_77
            +1 javadoc 1m 55s trunk passed with JDK v1.7.0_95
            +1 mvninstall 0m 48s the patch passed
            +1 compile 0m 49s the patch passed with JDK v1.8.0_77
            +1 javac 0m 49s the patch passed
            +1 compile 0m 43s the patch passed with JDK v1.7.0_95
            +1 javac 0m 43s the patch passed
            +1 checkstyle 0m 22s hadoop-hdfs-project/hadoop-hdfs: patch generated 0 new + 392 unchanged - 3 fixed = 392 total (was 395)
            +1 mvnsite 0m 51s the patch passed
            +1 mvneclipse 0m 12s the patch passed
            +1 whitespace 0m 0s Patch has no whitespace issues.
            +1 xml 0m 0s The patch has no ill-formed XML file.
            +1 findbugs 2m 9s the patch passed
            +1 javadoc 1m 14s the patch passed with JDK v1.8.0_77
            +1 javadoc 1m 50s the patch passed with JDK v1.7.0_95
            -1 unit 90m 51s hadoop-hdfs in the patch failed with JDK v1.8.0_77.
            -1 unit 78m 59s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
            +1 asflicense 0m 20s Patch does not generate ASF License warnings.
            197m 1s



            Reason Tests
            JDK v1.8.0_77 Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer
              hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
              hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
              hadoop.hdfs.TestFileAppend
              hadoop.hdfs.server.datanode.TestDataNodeMXBean
              hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
              hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs
            JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage
              hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
              hadoop.hdfs.server.namenode.TestReconstructStripedBlocks
              hadoop.hdfs.server.namenode.TestEditLog
              hadoop.hdfs.server.datanode.TestDirectoryScanner



            Subsystem Report/Notes
            Docker Image:yetus/hadoop:fbe3e86
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799112/HDFS-10297.003.patch
            JIRA Issue HDFS-10297
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
            uname Linux 24b71fe7d361 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
            git revision trunk / cab9cba
            Default Java 1.7.0_95
            Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
            findbugs v3.0.0
            unit https://builds.apache.org/job/PreCommit-HDFS-Build/15180/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt
            unit https://builds.apache.org/job/PreCommit-HDFS-Build/15180/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
            unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15180/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15180/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
            JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15180/testReport/
            modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
            Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15180/console
            Powered by Apache Yetus 0.2.0 http://yetus.apache.org

            This message was automatically generated.

            Show
            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 9s trunk passed +1 compile 0m 52s trunk passed with JDK v1.8.0_77 +1 compile 0m 44s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 24s trunk passed +1 mvnsite 0m 53s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 2m 0s trunk passed +1 javadoc 1m 14s trunk passed with JDK v1.8.0_77 +1 javadoc 1m 55s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 48s the patch passed +1 compile 0m 49s the patch passed with JDK v1.8.0_77 +1 javac 0m 49s the patch passed +1 compile 0m 43s the patch passed with JDK v1.7.0_95 +1 javac 0m 43s the patch passed +1 checkstyle 0m 22s hadoop-hdfs-project/hadoop-hdfs: patch generated 0 new + 392 unchanged - 3 fixed = 392 total (was 395) +1 mvnsite 0m 51s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 2m 9s the patch passed +1 javadoc 1m 14s the patch passed with JDK v1.8.0_77 +1 javadoc 1m 50s the patch passed with JDK v1.7.0_95 -1 unit 90m 51s hadoop-hdfs in the patch failed with JDK v1.8.0_77. -1 unit 78m 59s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 20s Patch does not generate ASF License warnings. 197m 1s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl   hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.TestFileAppend   hadoop.hdfs.server.datanode.TestDataNodeMXBean   hadoop.hdfs.server.blockmanagement.TestReplicationPolicy   hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.server.namenode.TestReconstructStripedBlocks   hadoop.hdfs.server.namenode.TestEditLog   hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799112/HDFS-10297.003.patch JIRA Issue HDFS-10297 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 24b71fe7d361 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / cab9cba Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15180/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15180/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15180/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15180/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15180/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15180/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
            Hide
            Permalink
            jzhuge John Zhuge added a comment -

            Could someone kindly review this simple patch?

            Show
            jzhuge John Zhuge added a comment - Could someone kindly review this simple patch?
            Hide
            Permalink
            andrew.wang Andrew Wang added a comment -

            LGTM +1, will commit shortly.

            Show
            andrew.wang Andrew Wang added a comment - LGTM +1, will commit shortly.
            Hide
            Permalink
            andrew.wang Andrew Wang added a comment -

            Committed to trunk and branch-2, branch-2.8 wasn't clean. John, if you want it in 2.8, do you mind providing a branch-2.8 patch? Else we can just resolve.

            Show
            andrew.wang Andrew Wang added a comment - Committed to trunk and branch-2, branch-2.8 wasn't clean. John, if you want it in 2.8, do you mind providing a branch-2.8 patch? Else we can just resolve.
            Hide
            Permalink
            hudson Hudson added a comment -

            FAILURE: Integrated in Hadoop-trunk-Commit #9672 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9672/)
            HDFS-10297. Increase default balance bandwidth and concurrent moves. (wang: rev 6be22ddbf1b6f3c19ec70c63ddb8f5519d18dd72)

            • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
            • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
            Show
            hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9672 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9672/ ) HDFS-10297 . Increase default balance bandwidth and concurrent moves. (wang: rev 6be22ddbf1b6f3c19ec70c63ddb8f5519d18dd72) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
            Hide
            Permalink
            jzhuge John Zhuge added a comment -

            Thank you Andrew Wang. I will create a patch for 2.8.

            Show
            jzhuge John Zhuge added a comment - Thank you Andrew Wang . I will create a patch for 2.8.
            Hide
            Permalink
            jzhuge John Zhuge added a comment -

            Wow, so many properties of file hdfs-default.xml in branch-2 (line 2972 to 3984) are NOT in branch-2.8 or 2.7 including dfs.datanode.balance.max.concurrent.moves?!

            Show
            jzhuge John Zhuge added a comment - Wow, so many properties of file hdfs-default.xml in branch-2 (line 2972 to 3984) are NOT in branch-2.8 or 2.7 including dfs.datanode.balance.max.concurrent.moves ?!
            Hide
            Permalink
            jzhuge John Zhuge added a comment -

            Ray Chiang's HDFS-8356 added the missing properties. Is it applicable to 2.8 and 2.7? My guess is no.

            Show
            jzhuge John Zhuge added a comment - Ray Chiang 's HDFS-8356 added the missing properties. Is it applicable to 2.8 and 2.7? My guess is no.
            Hide
            Permalink
            rchiang Ray Chiang added a comment -

            Yeah, I only did trunk and branch-2 versions for HDFS-8356. I didn't go backwards compatible with all the branch-2 minor versions.

            Show
            rchiang Ray Chiang added a comment - Yeah, I only did trunk and branch-2 versions for HDFS-8356 . I didn't go backwards compatible with all the branch-2 minor versions.
            Hide
            Permalink
            jzhuge John Zhuge added a comment -

            Thanks Ray Chiang.

            Show
            jzhuge John Zhuge added a comment - Thanks Ray Chiang .
            Hide
            Permalink
            jzhuge John Zhuge added a comment -

            Upload HDFS-10297.003.branch-2.8.patch for branch 2.8.

            Show
            jzhuge John Zhuge added a comment - Upload HDFS-10297 .003.branch-2.8.patch for branch 2.8.
            Hide
            Permalink
            jzhuge John Zhuge added a comment -

            Andrew Wang, I uploaded a patch for branch-2.8, please merge.

            Show
            jzhuge John Zhuge added a comment - Andrew Wang , I uploaded a patch for branch-2.8, please merge.
            Hide
            Permalink
            andrew.wang Andrew Wang added a comment -

            Pushed, thanks John!

            Show
            andrew.wang Andrew Wang added a comment - Pushed, thanks John!

              People

              • Assignee:
                jzhuge John Zhuge
                Reporter:
                jzhuge John Zhuge
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development