Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Steps to reconfigure:
      1. change value of the parameter in corresponding xml configuration file
      2. to reconfigure, run
          hdfs dfsadmin -reconfig datanode <dn_addr>:<ipc_port> start
      3. repeat step 2 until all DNs are reconfigured
      4. to check status of the most recent reconfigure operation, run
          hdfs dfsadmin -reconfig datanode <dn_addr>:<ipc_port> status
      5. to query a list reconfigurable properties on DN, run
          hdfs dfsadmin -reconfig datanode <dn_addr>:<ipc_port> properties
      Show
      Steps to reconfigure: 1. change value of the parameter in corresponding xml configuration file 2. to reconfigure, run     hdfs dfsadmin -reconfig datanode <dn_addr>:<ipc_port> start 3. repeat step 2 until all DNs are reconfigured 4. to check status of the most recent reconfigure operation, run     hdfs dfsadmin -reconfig datanode <dn_addr>:<ipc_port> status 5. to query a list reconfigurable properties on DN, run     hdfs dfsadmin -reconfig datanode <dn_addr>:<ipc_port> properties

      Description

      This is to reconfigure

      dfs.datanode.balance.max.concurrent.moves
      

      without restarting DN.

      1. HDFS-9214.001.patch
        12 kB
        Xiaobing Zhou
      2. HDFS-9214.002.patch
        17 kB
        Xiaobing Zhou
      3. HDFS-9214.003.patch
        15 kB
        Xiaobing Zhou
      4. HDFS-9214.004.patch
        16 kB
        Xiaobing Zhou
      5. HDFS-9214.005.patch
        17 kB
        Xiaobing Zhou

        Issue Links

          Activity

          Hide
          xiaobingo Xiaobing Zhou added a comment -

          I posted the 001 patch with a hope someone can review it. Thanks.

          Show
          xiaobingo Xiaobing Zhou added a comment - I posted the 001 patch with a hope someone can review it. Thanks.
          Hide
          xiaobingo Xiaobing Zhou added a comment -

          Patch 002 with more tests added.

          Show
          xiaobingo Xiaobing Zhou added a comment - Patch 002 with more tests added.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Hi Xiaobing Zhou, the change looks good, one suggestion below.

          DataXceiverServer does not need to extend ReconfigurableBase, since the interface between DataNode and DataXceiverServer is private. Then you can eliminate DataXceiverServer#getReconfigurableProperties and #getNewConf as no one should be invoking these functions.

          Show
          arpitagarwal Arpit Agarwal added a comment - Hi Xiaobing Zhou , the change looks good, one suggestion below. DataXceiverServer does not need to extend ReconfigurableBase, since the interface between DataNode and DataXceiverServer is private. Then you can eliminate DataXceiverServer#getReconfigurableProperties and #getNewConf as no one should be invoking these functions.
          Hide
          xiaobingo Xiaobing Zhou added a comment -

          Thanks for Arpit Agarwal's reviews. I posted v003 patch to address the issuess, kindly review!

          Show
          xiaobingo Xiaobing Zhou added a comment - Thanks for Arpit Agarwal 's reviews. I posted v003 patch to address the issuess, kindly review!
          Hide
          liuml07 Mingliang Liu added a comment -

          Overall looks good. Minor comments:

          1. Any logging when re-configure properties in DataNode will be helpful
          2. Make DataXceiverServer#maxThreads final if possible
          3. Annotate getMaxThreads() with @VisiableForTesting
          Show
          liuml07 Mingliang Liu added a comment - Overall looks good. Minor comments: Any logging when re-configure properties in DataNode will be helpful Make DataXceiverServer#maxThreads final if possible Annotate getMaxThreads() with @VisiableForTesting
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 16s trunk passed
          +1 compile 0m 37s trunk passed with JDK v1.8.0_66
          +1 compile 0m 39s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 15s trunk passed
          +1 mvnsite 0m 52s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 52s trunk passed
          +1 javadoc 1m 8s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 41s trunk passed with JDK v1.7.0_85
          +1 mvninstall 0m 47s the patch passed
          +1 compile 0m 43s the patch passed with JDK v1.8.0_66
          +1 javac 0m 43s the patch passed
          +1 compile 0m 40s the patch passed with JDK v1.7.0_85
          +1 javac 0m 40s the patch passed
          -1 checkstyle 0m 16s Patch generated 11 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 157, now 167).
          +1 mvnsite 0m 52s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 findbugs 2m 5s hadoop-hdfs-project/hadoop-hdfs introduced 1 new FindBugs issues.
          +1 javadoc 1m 6s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 46s the patch passed with JDK v1.7.0_85
          -1 unit 50m 33s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 51m 14s hadoop-hdfs in the patch failed with JDK v1.7.0_85.
          -1 asflicense 0m 22s Patch generated 59 ASF License warnings.
          128m 54s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            Boxing/unboxing to parse a primitive org.apache.hadoop.hdfs.server.datanode.DataNode.reconfigurePropertyImpl(String, String) At DataNode.java:org.apache.hadoop.hdfs.server.datanode.DataNode.reconfigurePropertyImpl(String, String) At DataNode.java:[line 537]
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestAclsEndToEnd
            hadoop.hdfs.tools.TestDFSAdmin
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.TestRollingUpgrade
            hadoop.hdfs.TestDFSClientExcludedNodes
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
          JDK v1.7.0_85 Failed junit tests hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
            hadoop.hdfs.tools.TestDFSAdmin
            hadoop.hdfs.TestReadStripedFileWithDecoding
            hadoop.hdfs.TestPread



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774443/HDFS-9214.003.patch
          JIRA Issue HDFS-9214
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6c66751bb4e2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d57fd18
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13670/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13670/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13670/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13670/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13670/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13670/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13670/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13670/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 75MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13670/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 16s trunk passed +1 compile 0m 37s trunk passed with JDK v1.8.0_66 +1 compile 0m 39s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 52s trunk passed +1 javadoc 1m 8s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 41s trunk passed with JDK v1.7.0_85 +1 mvninstall 0m 47s the patch passed +1 compile 0m 43s the patch passed with JDK v1.8.0_66 +1 javac 0m 43s the patch passed +1 compile 0m 40s the patch passed with JDK v1.7.0_85 +1 javac 0m 40s the patch passed -1 checkstyle 0m 16s Patch generated 11 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 157, now 167). +1 mvnsite 0m 52s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 2m 5s hadoop-hdfs-project/hadoop-hdfs introduced 1 new FindBugs issues. +1 javadoc 1m 6s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 46s the patch passed with JDK v1.7.0_85 -1 unit 50m 33s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 51m 14s hadoop-hdfs in the patch failed with JDK v1.7.0_85. -1 asflicense 0m 22s Patch generated 59 ASF License warnings. 128m 54s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   Boxing/unboxing to parse a primitive org.apache.hadoop.hdfs.server.datanode.DataNode.reconfigurePropertyImpl(String, String) At DataNode.java:org.apache.hadoop.hdfs.server.datanode.DataNode.reconfigurePropertyImpl(String, String) At DataNode.java: [line 537] JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestAclsEndToEnd   hadoop.hdfs.tools.TestDFSAdmin   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.TestDFSClientExcludedNodes   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure JDK v1.7.0_85 Failed junit tests hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.tools.TestDFSAdmin   hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.TestPread Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774443/HDFS-9214.003.patch JIRA Issue HDFS-9214 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6c66751bb4e2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d57fd18 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13670/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13670/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/13670/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13670/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13670/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13670/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13670/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13670/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 75MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13670/console This message was automatically generated.
          Hide
          xiaobingo Xiaobing Zhou added a comment -

          All fixed in v004. Thanks!

          Show
          xiaobingo Xiaobing Zhou added a comment - All fixed in v004. Thanks!
          Hide
          xiaobingo Xiaobing Zhou added a comment -

          v004 fixed Mingliang Liu's comment, check-style issues and one findbugs warning.

          Show
          xiaobingo Xiaobing Zhou added a comment - v004 fixed Mingliang Liu 's comment, check-style issues and one findbugs warning.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks for the v4 patch Xiaobing Zhou

          +1 pending Jenkins. Will hold off committing until tomorrow in case Mingliang Liu wants to take another look.

          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks for the v4 patch Xiaobing Zhou +1 pending Jenkins. Will hold off committing until tomorrow in case Mingliang Liu wants to take another look.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 9m 9s trunk passed
          +1 compile 0m 55s trunk passed with JDK v1.8.0_66
          +1 compile 0m 50s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 19s trunk passed
          +1 mvnsite 1m 4s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 2m 19s trunk passed
          +1 javadoc 1m 23s trunk passed with JDK v1.8.0_66
          +1 javadoc 2m 12s trunk passed with JDK v1.7.0_85
          +1 mvninstall 1m 0s the patch passed
          +1 compile 0m 55s the patch passed with JDK v1.8.0_66
          +1 javac 0m 55s the patch passed
          +1 compile 0m 51s the patch passed with JDK v1.7.0_85
          +1 javac 0m 51s the patch passed
          -1 checkstyle 0m 19s Patch generated 3 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 157, now 142).
          +1 mvnsite 1m 4s the patch passed
          +1 mvneclipse 0m 17s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 29s the patch passed
          +1 javadoc 1m 22s the patch passed with JDK v1.8.0_66
          +1 javadoc 2m 11s the patch passed with JDK v1.7.0_85
          -1 unit 68m 1s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 65m 29s hadoop-hdfs in the patch failed with JDK v1.7.0_85.
          -1 asflicense 0m 22s Patch generated 59 ASF License warnings.
          166m 1s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
            hadoop.hdfs.TestDFSClientRetries
            hadoop.hdfs.tools.TestDFSAdmin
            hadoop.hdfs.server.datanode.TestBlockScanner
          JDK v1.7.0_85 Failed junit tests hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.tools.TestDFSAdmin
            hadoop.hdfs.TestBlockStoragePolicy



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775624/HDFS-9214.004.patch
          JIRA Issue HDFS-9214
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 8e60cf77ba71 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 62e9348
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13756/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13756/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13756/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13756/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13756/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13756/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13756/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13756/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 9m 9s trunk passed +1 compile 0m 55s trunk passed with JDK v1.8.0_66 +1 compile 0m 50s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 19s trunk passed +1 mvnsite 1m 4s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 2m 19s trunk passed +1 javadoc 1m 23s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 12s trunk passed with JDK v1.7.0_85 +1 mvninstall 1m 0s the patch passed +1 compile 0m 55s the patch passed with JDK v1.8.0_66 +1 javac 0m 55s the patch passed +1 compile 0m 51s the patch passed with JDK v1.7.0_85 +1 javac 0m 51s the patch passed -1 checkstyle 0m 19s Patch generated 3 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 157, now 142). +1 mvnsite 1m 4s the patch passed +1 mvneclipse 0m 17s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 29s the patch passed +1 javadoc 1m 22s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 11s the patch passed with JDK v1.7.0_85 -1 unit 68m 1s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 65m 29s hadoop-hdfs in the patch failed with JDK v1.7.0_85. -1 asflicense 0m 22s Patch generated 59 ASF License warnings. 166m 1s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040   hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport   hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.tools.TestDFSAdmin   hadoop.hdfs.server.datanode.TestBlockScanner JDK v1.7.0_85 Failed junit tests hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.tools.TestDFSAdmin   hadoop.hdfs.TestBlockStoragePolicy Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775624/HDFS-9214.004.patch JIRA Issue HDFS-9214 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8e60cf77ba71 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 62e9348 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13756/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13756/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13756/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13756/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13756/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13756/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13756/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13756/console This message was automatically generated.
          Hide
          liuml07 Mingliang Liu added a comment -

          The latest patch looks good to me. Thanks.

          Show
          liuml07 Mingliang Liu added a comment - The latest patch looks good to me. Thanks.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          The only related test failure is TestDFSAdmin#testGetReconfigAllowedProperties.

          Show
          arpitagarwal Arpit Agarwal added a comment - The only related test failure is TestDFSAdmin#testGetReconfigAllowedProperties.
          Hide
          xiaobingo Xiaobing Zhou added a comment -

          V005 fixed this one. It fails because one more DataNode parameter is reconfigurable. Thanks!

          Show
          xiaobingo Xiaobing Zhou added a comment - V005 fixed this one. It fails because one more DataNode parameter is reconfigurable. Thanks!
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          +1 for v005 patch pending Jenkins. Thanks Xiaobing Zhou.

          We can probably remove this test in a future checkin as targeted unit tests for each reconfigurable property exist.

          Show
          arpitagarwal Arpit Agarwal added a comment - +1 for v005 patch pending Jenkins. Thanks Xiaobing Zhou . We can probably remove this test in a future checkin as targeted unit tests for each reconfigurable property exist.
          Hide
          xiaobingo Xiaobing Zhou added a comment -

          Reasonable. Let's do that for other similar patches.

          Show
          xiaobingo Xiaobing Zhou added a comment - Reasonable. Let's do that for other similar patches.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 9m 0s trunk passed
          +1 compile 0m 54s trunk passed with JDK v1.8.0_66
          +1 compile 0m 52s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 19s trunk passed
          +1 mvnsite 1m 4s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 2m 18s trunk passed
          +1 javadoc 1m 22s trunk passed with JDK v1.8.0_66
          +1 javadoc 2m 6s trunk passed with JDK v1.7.0_85
          +1 mvninstall 0m 58s the patch passed
          +1 compile 0m 52s the patch passed with JDK v1.8.0_66
          +1 javac 0m 52s the patch passed
          +1 compile 0m 51s the patch passed with JDK v1.7.0_85
          +1 javac 0m 51s the patch passed
          -1 checkstyle 0m 18s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 157, now 140).
          +1 mvnsite 1m 0s the patch passed
          +1 mvneclipse 0m 16s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 30s the patch passed
          +1 javadoc 1m 22s the patch passed with JDK v1.8.0_66
          +1 javadoc 2m 10s the patch passed with JDK v1.7.0_85
          -1 unit 70m 9s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 62m 36s hadoop-hdfs in the patch failed with JDK v1.7.0_85.
          -1 asflicense 0m 20s Patch generated 59 ASF License warnings.
          164m 41s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.TestMiniDFSCluster
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.server.namenode.TestCacheDirectives
            hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
            hadoop.hdfs.server.datanode.TestBlockScanner
          JDK v1.7.0_85 Failed junit tests hadoop.hdfs.server.namenode.TestFSEditLogLoader
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.tools.TestJMXGet
            hadoop.hdfs.TestDFSUpgradeFromImage



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775809/HDFS-9214.005.patch
          JIRA Issue HDFS-9214
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux a51f134a53ba 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3fa33b5
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13769/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13769/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13769/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13769/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13769/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13769/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13769/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13769/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 9m 0s trunk passed +1 compile 0m 54s trunk passed with JDK v1.8.0_66 +1 compile 0m 52s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 19s trunk passed +1 mvnsite 1m 4s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 2m 18s trunk passed +1 javadoc 1m 22s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 6s trunk passed with JDK v1.7.0_85 +1 mvninstall 0m 58s the patch passed +1 compile 0m 52s the patch passed with JDK v1.8.0_66 +1 javac 0m 52s the patch passed +1 compile 0m 51s the patch passed with JDK v1.7.0_85 +1 javac 0m 51s the patch passed -1 checkstyle 0m 18s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 157, now 140). +1 mvnsite 1m 0s the patch passed +1 mvneclipse 0m 16s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 30s the patch passed +1 javadoc 1m 22s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 10s the patch passed with JDK v1.7.0_85 -1 unit 70m 9s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 62m 36s hadoop-hdfs in the patch failed with JDK v1.7.0_85. -1 asflicense 0m 20s Patch generated 59 ASF License warnings. 164m 41s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.TestMiniDFSCluster   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.namenode.TestCacheDirectives   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport   hadoop.hdfs.server.datanode.TestBlockScanner JDK v1.7.0_85 Failed junit tests hadoop.hdfs.server.namenode.TestFSEditLogLoader   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.tools.TestJMXGet   hadoop.hdfs.TestDFSUpgradeFromImage Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775809/HDFS-9214.005.patch JIRA Issue HDFS-9214 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a51f134a53ba 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3fa33b5 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13769/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13769/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13769/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13769/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13769/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13769/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13769/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13769/console This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment - - edited

          I've committed this. Thanks for the contribution Xiaobing Zhou. Verified that test failures are unrelated.

          Show
          arpitagarwal Arpit Agarwal added a comment - - edited I've committed this. Thanks for the contribution Xiaobing Zhou . Verified that test failures are unrelated.
          Hide
          cnauroth Chris Nauroth added a comment -

          There is a license warning on the new TestDataNodeReconfiguration class. Would you mind adding the license header there?

          The warning was easy to miss, because we currently have a problem with tests writing files outside the Maven target directory, which then get flagged for license violations. TestDataNodeReconfiguration was at the bottom of the list.

          https://builds.apache.org/job/PreCommit-HDFS-Build/13769/artifact/patchprocess/patch-asflicense-problems.txt

          HDFS-9263 tracks the effort to fix that.

          Show
          cnauroth Chris Nauroth added a comment - There is a license warning on the new TestDataNodeReconfiguration class. Would you mind adding the license header there? The warning was easy to miss, because we currently have a problem with tests writing files outside the Maven target directory, which then get flagged for license violations. TestDataNodeReconfiguration was at the bottom of the list. https://builds.apache.org/job/PreCommit-HDFS-Build/13769/artifact/patchprocess/patch-asflicense-problems.txt HDFS-9263 tracks the effort to fix that.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8925 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8925/)
          HDFS-9214. Support reconfiguring (arp: rev 9d817fa1b14b477e5440ae4edd78de849976d9b5)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeReconfiguration.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSAdmin.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8925 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8925/ ) HDFS-9214 . Support reconfiguring (arp: rev 9d817fa1b14b477e5440ae4edd78de849976d9b5) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeReconfiguration.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSAdmin.java
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks for the catch Chris. Pushed commits to add the missing license header.

          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks for the catch Chris. Pushed commits to add the missing license header.
          Hide
          cnauroth Chris Nauroth added a comment -

          Thank you, Arpit.

          Show
          cnauroth Chris Nauroth added a comment - Thank you, Arpit.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8926 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8926/)
          HDFS-9214. Add missing license header (arp: rev bc6d8d981477c1d8d6cff19f2b6fc01676b4d332)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeReconfiguration.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8926 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8926/ ) HDFS-9214 . Add missing license header (arp: rev bc6d8d981477c1d8d6cff19f2b6fc01676b4d332) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeReconfiguration.java
          Hide
          hudson Hudson added a comment -

          ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #667 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/667/)
          HDFS-9214. Support reconfiguring (arp: rev 9d817fa1b14b477e5440ae4edd78de849976d9b5)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSAdmin.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeReconfiguration.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            HDFS-9214. Add missing license header (arp: rev bc6d8d981477c1d8d6cff19f2b6fc01676b4d332)
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeReconfiguration.java
          Show
          hudson Hudson added a comment - ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #667 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/667/ ) HDFS-9214 . Support reconfiguring (arp: rev 9d817fa1b14b477e5440ae4edd78de849976d9b5) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSAdmin.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeReconfiguration.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt HDFS-9214 . Add missing license header (arp: rev bc6d8d981477c1d8d6cff19f2b6fc01676b4d332) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeReconfiguration.java
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Please add release note to describe how to reconfigure dfs.datanode.balance.max.concurrent.moves. Thanks.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Please add release note to describe how to reconfigure dfs.datanode.balance.max.concurrent.moves. Thanks.
          Hide
          xiaobingo Xiaobing Zhou added a comment -

          Thanks Tsz Wo Nicholas Sze for bringing this up. The release note has been added.

          Show
          xiaobingo Xiaobing Zhou added a comment - Thanks Tsz Wo Nicholas Sze for bringing this up. The release note has been added.

            People

            • Assignee:
              xiaobingo Xiaobing Zhou
              Reporter:
              xiaobingo Xiaobing Zhou
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development