Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9842

dfs.datanode.balance.bandwidthPerSec should accept friendly size units

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: balancer & mover
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In HDFS-1314, it made the dfs.blocksize can accept different size type like mb, kb, gb, etc. Using "dfs.block.size=8388608" works and "dfs.block.size=8mb" also can work, and this type will finally transform to byte format. So we also can make similar property dfs.datanode.balance.bandwidthPerSec be friendly accepted in size unit.

        Issue Links

          Activity

          Hide
          linyiqun Yiqun Lin added a comment -

          Thanks @Arpit Agarwal for quick review and commit!

          Show
          linyiqun Yiqun Lin added a comment - Thanks @Arpit Agarwal for quick review and commit!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9341 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9341/)
          HDFS-9842. dfs.datanode.balance.bandwidthPerSec should accept friendly (arp: rev a63301a8645ca9ab36b782d03af96e9c4e901db5)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9341 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9341/ ) HDFS-9842 . dfs.datanode.balance.bandwidthPerSec should accept friendly (arp: rev a63301a8645ca9ab36b782d03af96e9c4e901db5) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Committed for 2.8.0.

          Show
          arpitagarwal Arpit Agarwal added a comment - Committed for 2.8.0.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks for the contribution Yiqun Lin. +1 for the patch.

          The test failures look unrelated. I will commit it shortly.

          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks for the contribution Yiqun Lin . +1 for the patch. The test failures look unrelated. I will commit it shortly.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 25s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 8m 53s trunk passed
          +1 compile 0m 55s trunk passed with JDK v1.8.0_72
          +1 compile 0m 47s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 23s trunk passed
          +1 mvnsite 1m 0s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 2m 1s trunk passed
          +1 javadoc 1m 21s trunk passed with JDK v1.8.0_72
          +1 javadoc 2m 0s trunk passed with JDK v1.7.0_95
          +1 mvninstall 1m 2s the patch passed
          +1 compile 0m 53s the patch passed with JDK v1.8.0_72
          +1 javac 0m 53s the patch passed
          +1 compile 0m 40s the patch passed with JDK v1.7.0_95
          +1 javac 0m 40s the patch passed
          -1 checkstyle 0m 18s hadoop-hdfs-project/hadoop-hdfs: patch generated 1 new + 14 unchanged - 0 fixed = 15 total (was 14)
          +1 mvnsite 0m 53s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 2m 12s the patch passed
          +1 javadoc 1m 11s the patch passed with JDK v1.8.0_72
          +1 javadoc 2m 11s the patch passed with JDK v1.7.0_95
          -1 unit 94m 34s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
          -1 unit 91m 14s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 20s Patch does not generate ASF License warnings.
          216m 2s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.TestLocalDFS
            hadoop.hdfs.TestFileAppend
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.server.namenode.ha.TestHAAppend
            hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles
            hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs
            hadoop.hdfs.qjournal.client.TestQuorumJournalManager
            hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer
            hadoop.hdfs.server.datanode.TestDirectoryScanner



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788996/HDFS-9842.001.patch
          JIRA Issue HDFS-9842
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 4a9eed46e89c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 5e7d4d5
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14559/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14559/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14559/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14559/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14559/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14559/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14559/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 25s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 53s trunk passed +1 compile 0m 55s trunk passed with JDK v1.8.0_72 +1 compile 0m 47s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 23s trunk passed +1 mvnsite 1m 0s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 2m 1s trunk passed +1 javadoc 1m 21s trunk passed with JDK v1.8.0_72 +1 javadoc 2m 0s trunk passed with JDK v1.7.0_95 +1 mvninstall 1m 2s the patch passed +1 compile 0m 53s the patch passed with JDK v1.8.0_72 +1 javac 0m 53s the patch passed +1 compile 0m 40s the patch passed with JDK v1.7.0_95 +1 javac 0m 40s the patch passed -1 checkstyle 0m 18s hadoop-hdfs-project/hadoop-hdfs: patch generated 1 new + 14 unchanged - 0 fixed = 15 total (was 14) +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 2m 12s the patch passed +1 javadoc 1m 11s the patch passed with JDK v1.8.0_72 +1 javadoc 2m 11s the patch passed with JDK v1.7.0_95 -1 unit 94m 34s hadoop-hdfs in the patch failed with JDK v1.8.0_72. -1 unit 91m 14s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 20s Patch does not generate ASF License warnings. 216m 2s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestLocalDFS   hadoop.hdfs.TestFileAppend   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles   hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs   hadoop.hdfs.qjournal.client.TestQuorumJournalManager   hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer   hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788996/HDFS-9842.001.patch JIRA Issue HDFS-9842 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 4a9eed46e89c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5e7d4d5 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14559/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14559/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14559/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14559/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14559/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14559/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14559/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          linyiqun Yiqun Lin added a comment -

          Attach a initial patch, kindly review, thanks.

          Show
          linyiqun Yiqun Lin added a comment - Attach a initial patch, kindly review, thanks.

            People

            • Assignee:
              linyiqun Yiqun Lin
              Reporter:
              linyiqun Yiqun Lin
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development