Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: balancer & mover
    • Labels:
      None

      Description

      Since HDFS-10813 has committed to the trunk, then we can use getNodes method to parse the nodes string and support multiple nodes with hdfs diskbalancer subcommands(ex -report, -query). In this JIRA, we are focusing on the subcommand -report.

      That means we can use command hdfs diskbalancer -report -node to print one or one more datanodes report info. A test input command(here I use UUID to specify one datanode):

      hdfs diskbalancer -report -node e05ade8e-fb28-42cf-9aa9-43e564c0ec99,38714337-84fb-4e35-9ea3-0bb47d6da700
      
      1. HDFS-10821.003.patch
        14 kB
        Yiqun Lin
      2. HDFS-10821.002.patch
        10 kB
        Yiqun Lin
      3. HDFS-10821.001.patch
        8 kB
        Yiqun Lin

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10427 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10427/)
          HDFS-10821. DiskBalancer: Report command support with multiple nodes. (aengineer: rev 8a93f45a80932a1ef62a6c20551e8cab95888fee)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSDiskbalancer.md
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/diskbalancer/command/TestDiskBalancerCommand.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/DiskBalancerException.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/Command.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/ReportCommand.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10427 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10427/ ) HDFS-10821 . DiskBalancer: Report command support with multiple nodes. (aengineer: rev 8a93f45a80932a1ef62a6c20551e8cab95888fee) (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSDiskbalancer.md (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/diskbalancer/command/TestDiskBalancerCommand.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/DiskBalancerException.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/Command.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/ReportCommand.java
          Hide
          anu Anu Engineer added a comment -

          Yiqun Lin Thank you for the contribution. I have committed this to trunk.

          Show
          anu Anu Engineer added a comment - Yiqun Lin Thank you for the contribution. I have committed this to trunk.
          Hide
          anu Anu Engineer added a comment -

          +1, LGTM. Yiqun Lin Thanks for updating the patch. I will commit this shortly.

          Show
          anu Anu Engineer added a comment - +1, LGTM. Yiqun Lin Thanks for updating the patch. I will commit this shortly.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 52s trunk passed
          +1 compile 1m 3s trunk passed
          +1 checkstyle 0m 32s trunk passed
          +1 mvnsite 1m 8s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 1m 54s trunk passed
          +1 javadoc 1m 5s trunk passed
          +1 mvninstall 0m 57s the patch passed
          +1 compile 0m 55s the patch passed
          +1 javac 0m 55s the patch passed
          +1 checkstyle 0m 27s the patch passed
          +1 mvnsite 1m 4s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 0s the patch passed
          +1 javadoc 1m 1s the patch passed
          -1 unit 78m 33s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          102m 11s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestFsDatasetCache
            hadoop.hdfs.TestRollingUpgrade



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10821
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827744/HDFS-10821.003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux c61a5b5bc8f8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / baab489
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16686/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16686/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16686/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 52s trunk passed +1 compile 1m 3s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 1m 8s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 1m 54s trunk passed +1 javadoc 1m 5s trunk passed +1 mvninstall 0m 57s the patch passed +1 compile 0m 55s the patch passed +1 javac 0m 55s the patch passed +1 checkstyle 0m 27s the patch passed +1 mvnsite 1m 4s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 0s the patch passed +1 javadoc 1m 1s the patch passed -1 unit 78m 33s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 102m 11s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestFsDatasetCache   hadoop.hdfs.TestRollingUpgrade Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10821 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827744/HDFS-10821.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c61a5b5bc8f8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / baab489 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16686/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16686/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16686/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 7s trunk passed
          +1 compile 0m 44s trunk passed
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 0m 50s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 40s trunk passed
          +1 javadoc 0m 57s trunk passed
          +1 mvninstall 0m 47s the patch passed
          +1 compile 0m 41s the patch passed
          +1 javac 0m 41s the patch passed
          +1 checkstyle 0m 23s the patch passed
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 48s the patch passed
          +1 javadoc 0m 51s the patch passed
          -1 unit 81m 27s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 30s The patch does not generate ASF License warnings.
          100m 59s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10821
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827739/HDFS-10821.003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6116f229ba07 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / baab489
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16685/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16685/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16685/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 7s trunk passed +1 compile 0m 44s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 50s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 40s trunk passed +1 javadoc 0m 57s trunk passed +1 mvninstall 0m 47s the patch passed +1 compile 0m 41s the patch passed +1 javac 0m 41s the patch passed +1 checkstyle 0m 23s the patch passed +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 48s the patch passed +1 javadoc 0m 51s the patch passed -1 unit 81m 27s hadoop-hdfs in the patch failed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 100m 59s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10821 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827739/HDFS-10821.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6116f229ba07 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / baab489 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16685/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16685/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16685/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          linyiqun Yiqun Lin added a comment - - edited

          ReUpload the v003 patch. I found there was no need to print the line Can't find a DataNode that matches '%s'. here if we have already printed the invalid nodes.

          Show
          linyiqun Yiqun Lin added a comment - - edited ReUpload the v003 patch. I found there was no need to print the line Can't find a DataNode that matches '%s'. here if we have already printed the invalid nodes.
          Hide
          linyiqun Yiqun Lin added a comment -

          Thanks a lot for the quick review, Anu Engineer.

          You might have to modify the getNodes function and return a list of invalid nodes or throw an exception and abort the whole operation,

          I prefer to throw an exception and abort the whole operation here. And in the future we can add a new option to ignore invalid nodes in input node string.

          The new output from the test testReportCommandWithInvalidNode:

          Processing report command
          Reporting volume information for DataNode(s) 'bb6d51c1-7345-42dc-aaa3-122e16e12320,invalidNode'.
          The node(s) 'invalidNode' not found. Please make sure that 'invalidNode' exists in the cluster.
          

          Attach a new patch, hope my latest patch can satisfy with you.

          Show
          linyiqun Yiqun Lin added a comment - Thanks a lot for the quick review, Anu Engineer . You might have to modify the getNodes function and return a list of invalid nodes or throw an exception and abort the whole operation, I prefer to throw an exception and abort the whole operation here. And in the future we can add a new option to ignore invalid nodes in input node string. The new output from the test testReportCommandWithInvalidNode : Processing report command Reporting volume information for DataNode(s) 'bb6d51c1-7345-42dc-aaa3-122e16e12320,invalidNode'. The node(s) 'invalidNode' not found. Please make sure that 'invalidNode' exists in the cluster. Attach a new patch, hope my latest patch can satisfy with you.
          Hide
          anu Anu Engineer added a comment -

          Yiqun Lin Thanks for quick turn around and the new patch. I really appreciate you adding the new test that tests the invalid node. Don't you think if we find an invalid node, we should really print out an error message on the console ?. I was thinking that output of the test for line 6,7 should be something like "$InvalidNode not found. Please make sure that $InvalidNode exists in the cluster." That way user is aware that he/she has made a mistake. You might have to modify the getNodes function and return a list of invalid nodes or throw an exception and abort the whole operation, I am fine with either option.

          I don't think empty lines indicate an error condition to the user, so while this assertion technically correct, I am worried it does not offer the best user experience.

          // The 7th line should not contain invalid node's info and
              // should be empty.
              assertTrue(!outputs.get(6).contains(invalidNode)
                  && outputs.get(6).isEmpty());
          

          This is what I see on my console, which is kind of hard to decode on a shell. I think the user experience would be a silent failure.

          outputs = {ArrayList@5593}  size = 7
           0 = "Processing report command"
           1 = "Reporting volume information for DataNode(s) '5c8a07de-7848-4c21-807f-ed6bba06c623,invalidNode'."
           2 = "127.0.0.1[127.0.0.1:50140] - <5c8a07de-7848-4c21-807f-ed6bba06c623>: 2 volumes with node data density 0.00."
           3 = "[DISK: volume-/Users/aengineer/diskBalancer/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data1] - 0.00 used: 4096/499077087232, 1.00 free: 499077083136/499077087232, isFailed: False, isReadOnly: False, isSkip: False, isTransient: False."
           4 = "[DISK: volume-/Users/aengineer/diskBalancer/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data2] - 0.00 used: 4096/499077087232, 1.00 free: 499077083136/499077087232, isFailed: False, isReadOnly: False, isSkip: False, isTransient: False."
           5 = ""
           6 = ""
          
          Show
          anu Anu Engineer added a comment - Yiqun Lin Thanks for quick turn around and the new patch. I really appreciate you adding the new test that tests the invalid node. Don't you think if we find an invalid node, we should really print out an error message on the console ?. I was thinking that output of the test for line 6,7 should be something like "$InvalidNode not found. Please make sure that $InvalidNode exists in the cluster." That way user is aware that he/she has made a mistake. You might have to modify the getNodes function and return a list of invalid nodes or throw an exception and abort the whole operation, I am fine with either option. I don't think empty lines indicate an error condition to the user, so while this assertion technically correct, I am worried it does not offer the best user experience. // The 7th line should not contain invalid node's info and // should be empty. assertTrue(!outputs.get(6).contains(invalidNode) && outputs.get(6).isEmpty()); This is what I see on my console, which is kind of hard to decode on a shell. I think the user experience would be a silent failure. outputs = {ArrayList@5593} size = 7 0 = "Processing report command" 1 = "Reporting volume information for DataNode(s) '5c8a07de-7848-4c21-807f-ed6bba06c623,invalidNode'." 2 = "127.0.0.1[127.0.0.1:50140] - <5c8a07de-7848-4c21-807f-ed6bba06c623>: 2 volumes with node data density 0.00." 3 = "[DISK: volume-/Users/aengineer/diskBalancer/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data1] - 0.00 used: 4096/499077087232, 1.00 free: 499077083136/499077087232, isFailed: False, isReadOnly: False, isSkip: False, isTransient: False." 4 = "[DISK: volume-/Users/aengineer/diskBalancer/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data2] - 0.00 used: 4096/499077087232, 1.00 free: 499077083136/499077087232, isFailed: False, isReadOnly: False, isSkip: False, isTransient: False." 5 = "" 6 = ""
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 39s trunk passed
          +1 compile 0m 49s trunk passed
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 0m 56s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 45s trunk passed
          +1 javadoc 1m 1s trunk passed
          +1 mvninstall 0m 52s the patch passed
          +1 compile 0m 48s the patch passed
          +1 javac 0m 48s the patch passed
          -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)
          +1 mvnsite 0m 50s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 48s the patch passed
          +1 javadoc 0m 59s the patch passed
          -1 unit 78m 42s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          100m 28s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.TestDecommission
            hadoop.hdfs.TestRollingUpgrade



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10821
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827477/HDFS-10821.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux d31565215e84 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d355573
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16665/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16665/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16665/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16665/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 39s trunk passed +1 compile 0m 49s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 56s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 45s trunk passed +1 javadoc 1m 1s trunk passed +1 mvninstall 0m 52s the patch passed +1 compile 0m 48s the patch passed +1 javac 0m 48s the patch passed -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) +1 mvnsite 0m 50s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 48s the patch passed +1 javadoc 0m 59s the patch passed -1 unit 78m 42s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 100m 28s Reason Tests Failed junit tests hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.TestDecommission   hadoop.hdfs.TestRollingUpgrade Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10821 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827477/HDFS-10821.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d31565215e84 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d355573 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16665/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16665/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16665/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16665/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          linyiqun Yiqun Lin added a comment -

          Thanks Anu Engineer for the review.

          so if you have an invalid node, along with valid nodes, I think we are going to ignore the invalid node silently.

          The method getNodes(nodeVal) will ignore the invalid node since the node is selected from the cluster rather than construct a new node. So if one node is a invalid node, the method DiskBalancerDataNode getNode(String nodeName) will return null and then it will be skipped.

          Could you please add a test case which passes an invalid node along with the valid set and make sure that report command detects the invalid node correctly.

          Done.

          Attach a new patch to address the comments, pending jenkins,

          Show
          linyiqun Yiqun Lin added a comment - Thanks Anu Engineer for the review. so if you have an invalid node, along with valid nodes, I think we are going to ignore the invalid node silently. The method getNodes(nodeVal) will ignore the invalid node since the node is selected from the cluster rather than construct a new node. So if one node is a invalid node, the method DiskBalancerDataNode getNode(String nodeName) will return null and then it will be skipped. Could you please add a test case which passes an invalid node along with the valid set and make sure that report command detects the invalid node correctly. Done. Attach a new patch to address the comments, pending jenkins,
          Hide
          anu Anu Engineer added a comment -

          Yiqun Lin Thank you for providing this patch. The changes look good, Some small comments.

          1. nit: Please remove "a" from this message.

          Print out volume information for a DataNode(s)
          

          2. Correct me if I am wrong, but nodeVal is a set of machines. so if you have an invalid node, along with valid nodes, I think we are going to ignore the invalid node silently.

          List<DiskBalancerDataNode> dbdns = getNodes(nodeVal);
          

          3. Could you please add a test case which passes an invalid node along with the valid set and make sure that report command detects the invalid node correctly.

          Show
          anu Anu Engineer added a comment - Yiqun Lin Thank you for providing this patch. The changes look good, Some small comments. 1. nit: Please remove "a" from this message. Print out volume information for a DataNode(s) 2. Correct me if I am wrong, but nodeVal is a set of machines. so if you have an invalid node, along with valid nodes, I think we are going to ignore the invalid node silently. List<DiskBalancerDataNode> dbdns = getNodes(nodeVal); 3. Could you please add a test case which passes an invalid node along with the valid set and make sure that report command detects the invalid node correctly.
          Hide
          linyiqun Yiqun Lin added a comment -

          Hi, Anu Engineer, can you take a quick look for the patch? I will post a new patch to fix checkstyle warnings and the comments from you if you have other comments for the latest patch.

          Show
          linyiqun Yiqun Lin added a comment - Hi, Anu Engineer , can you take a quick look for the patch? I will post a new patch to fix checkstyle warnings and the comments from you if you have other comments for the latest patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 43s trunk passed
          +1 compile 0m 45s trunk passed
          +1 checkstyle 0m 25s trunk passed
          +1 mvnsite 0m 49s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 40s trunk passed
          +1 javadoc 0m 54s trunk passed
          +1 mvninstall 0m 47s the patch passed
          +1 compile 0m 42s the patch passed
          +1 javac 0m 42s the patch passed
          -0 checkstyle 0m 22s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1)
          +1 mvnsite 0m 47s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 47s the patch passed
          +1 javadoc 0m 52s the patch passed
          -1 unit 72m 28s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          91m 17s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10821
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826382/HDFS-10821.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6c5e5f124970 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 20ae1fa
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16592/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16592/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16592/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16592/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 43s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 25s trunk passed +1 mvnsite 0m 49s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 40s trunk passed +1 javadoc 0m 54s trunk passed +1 mvninstall 0m 47s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed -0 checkstyle 0m 22s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1) +1 mvnsite 0m 47s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 47s the patch passed +1 javadoc 0m 52s the patch passed -1 unit 72m 28s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 91m 17s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10821 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826382/HDFS-10821.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6c5e5f124970 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 20ae1fa Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16592/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16592/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16592/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16592/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          linyiqun Yiqun Lin added a comment -

          Attach a initial patch. The documentation has been updated in the patch as well. The test output in my local:

          Processing report command
          Reporting volume information for DataNode(s) 'e05ade8e-fb28-42cf-9aa9-43e564c0ec99,38714337-84fb-4e35-9ea3-0bb47d6da700'.
          127.0.0.1[127.0.0.1:54994] - <38714337-84fb-4e35-9ea3-0bb47d6da700>: 2 volumes with node data density 0.00.
          [DISK: volume-/Users/lyq/Documents/work-project/hadoop-trunk/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data3] - 0.00 used: 4096/249821663232, 1.00 free: 249821659136/249821663232, isFailed: False, isReadOnly: False, isSkip: False, isTransient: False.
          [DISK: volume-/Users/lyq/Documents/work-project/hadoop-trunk/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data4] - 0.00 used: 4096/249821663232, 1.00 free: 249821659136/249821663232, isFailed: False, isReadOnly: False, isSkip: False, isTransient: False.
          
          127.0.0.1[127.0.0.1:54989] - <e05ade8e-fb28-42cf-9aa9-43e564c0ec99>: 2 volumes with node data density 0.00.
          [DISK: volume-/Users/lyq/Documents/work-project/hadoop-trunk/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data1] - 0.00 used: 4096/249821663232, 1.00 free: 249821659136/249821663232, isFailed: False, isReadOnly: False, isSkip: False, isTransient: False.
          [DISK: volume-/Users/lyq/Documents/work-project/hadoop-trunk/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data2] - 0.00 used: 4096/249821663232, 1.00 free: 249821659136/249821663232, isFailed: False, isReadOnly: False, isSkip: False, isTransient: False.
          
          Show
          linyiqun Yiqun Lin added a comment - Attach a initial patch. The documentation has been updated in the patch as well. The test output in my local: Processing report command Reporting volume information for DataNode(s) 'e05ade8e-fb28-42cf-9aa9-43e564c0ec99,38714337-84fb-4e35-9ea3-0bb47d6da700'. 127.0.0.1[127.0.0.1:54994] - <38714337-84fb-4e35-9ea3-0bb47d6da700>: 2 volumes with node data density 0.00. [DISK: volume-/Users/lyq/Documents/work-project/hadoop-trunk/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data3] - 0.00 used: 4096/249821663232, 1.00 free: 249821659136/249821663232, isFailed: False, isReadOnly: False, isSkip: False, isTransient: False. [DISK: volume-/Users/lyq/Documents/work-project/hadoop-trunk/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data4] - 0.00 used: 4096/249821663232, 1.00 free: 249821659136/249821663232, isFailed: False, isReadOnly: False, isSkip: False, isTransient: False. 127.0.0.1[127.0.0.1:54989] - <e05ade8e-fb28-42cf-9aa9-43e564c0ec99>: 2 volumes with node data density 0.00. [DISK: volume-/Users/lyq/Documents/work-project/hadoop-trunk/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data1] - 0.00 used: 4096/249821663232, 1.00 free: 249821659136/249821663232, isFailed: False, isReadOnly: False, isSkip: False, isTransient: False. [DISK: volume-/Users/lyq/Documents/work-project/hadoop-trunk/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data2] - 0.00 used: 4096/249821663232, 1.00 free: 249821659136/249821663232, isFailed: False, isReadOnly: False, isSkip: False, isTransient: False.

            People

            • Assignee:
              linyiqun Yiqun Lin
              Reporter:
              linyiqun Yiqun Lin
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development