Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11417

Add datanode admin command to get the storage info.

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.3
    • Fix Version/s: 2.9.0, 3.0.0-alpha4
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      It is good to add one admin command for datanode to get the data directory info like storage type, directory path, number of block, capacity, used space. This will be help full in large cluster where DN has multiple data directory configured.

      1. HDFS-11417.001.patch
        25 kB
        Surendra Singh Lilhore
      2. HDFS-11417.002.patch
        26 kB
        Surendra Singh Lilhore
      3. HDFS-11417.003.patch
        25 kB
        Surendra Singh Lilhore
      4. HDFS-11417.004.patch
        25 kB
        Surendra Singh Lilhore
      5. HDFS-11417.005.patch
        25 kB
        Surendra Singh Lilhore
      6. HDFS-11417.006.patch
        25 kB
        Surendra Singh Lilhore

        Issue Links

          Activity

          Hide
          kihwal Kihwal Lee added a comment -

          More info was added to the datanoed UI. HDFS-11375 is improving it. You might want to give them input if something you think is important is missing.

          Show
          kihwal Kihwal Lee added a comment - More info was added to the datanoed UI. HDFS-11375 is improving it. You might want to give them input if something you think is important is missing.
          Hide
          kihwal Kihwal Lee added a comment -

          Sorry, it is actually you who are working on both issues!

          Show
          kihwal Kihwal Lee added a comment - Sorry, it is actually you who are working on both issues!
          Hide
          surendrasingh Surendra Singh Lilhore added a comment - - edited

          I feel this is required when you have only shell interface to access the cluster data.

          Show
          surendrasingh Surendra Singh Lilhore added a comment - - edited I feel this is required when you have only shell interface to access the cluster data.
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Attached initial patch, Please review..
          Added new command

          ./hdfs dfsadmin -getStorageReport 127.0.0.1:9867
          Active Storage(3):
          
          Directory: /storage1/dfs/dn
          StorageType: SSD
          Capacity Used: 32768(32 KB)
          Capacity Left: 428230676480(398.82 GB)
          Capacity Reserved: 0(0 B)
          Reserved Space for Replicas: 0(0 B)
          Blocks: 0
          
          Directory: /storage2/dfs/dn
          StorageType: ARCHIVE
          Capacity Used: 32768(32 KB)
          Capacity Left: 428230676480(398.82 GB)
          Capacity Reserved: 0(0 B)
          Reserved Space for Replicas: 0(0 B)
          Blocks: 0
          
          Directory: /storage3/dfs/dn
          StorageType: DISK
          Capacity Used: 131072(128 KB)
          Capacity Left: 428230676480(398.82 GB)
          Capacity Reserved: 0(0 B)
          Reserved Space for Replicas: 0(0 B)
          Blocks: 6
          
          Show
          surendrasingh Surendra Singh Lilhore added a comment - Attached initial patch, Please review.. Added new command ./hdfs dfsadmin -getStorageReport 127.0.0.1:9867 Active Storage(3): Directory: /storage1/dfs/dn StorageType: SSD Capacity Used: 32768(32 KB) Capacity Left: 428230676480(398.82 GB) Capacity Reserved: 0(0 B) Reserved Space for Replicas: 0(0 B) Blocks: 0 Directory: /storage2/dfs/dn StorageType: ARCHIVE Capacity Used: 32768(32 KB) Capacity Left: 428230676480(398.82 GB) Capacity Reserved: 0(0 B) Reserved Space for Replicas: 0(0 B) Blocks: 0 Directory: /storage3/dfs/dn StorageType: DISK Capacity Used: 131072(128 KB) Capacity Left: 428230676480(398.82 GB) Capacity Reserved: 0(0 B) Reserved Space for Replicas: 0(0 B) Blocks: 6
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 7s Maven dependency ordering for branch
          +1 mvninstall 12m 39s trunk passed
          +1 compile 1m 24s trunk passed
          +1 checkstyle 0m 39s trunk passed
          +1 mvnsite 1m 22s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 3m 10s trunk passed
          +1 javadoc 1m 1s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 16s the patch passed
          +1 compile 1m 19s the patch passed
          +1 cc 1m 19s the patch passed
          +1 javac 1m 19s the patch passed
          +1 checkstyle 0m 36s the patch passed
          +1 mvnsite 1m 19s the patch passed
          +1 mvneclipse 0m 21s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 19s the patch passed
          +1 javadoc 0m 55s the patch passed
          +1 unit 0m 55s hadoop-hdfs-client in the patch passed.
          -1 unit 90m 13s hadoop-hdfs in the patch failed.
          -1 asflicense 0m 18s The patch generated 1 ASF License warnings.
          123m 6s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11417
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12854780/HDFS-11417.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux de6375b69899 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 05391c1
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18449/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18449/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/18449/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18449/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 7s Maven dependency ordering for branch +1 mvninstall 12m 39s trunk passed +1 compile 1m 24s trunk passed +1 checkstyle 0m 39s trunk passed +1 mvnsite 1m 22s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 3m 10s trunk passed +1 javadoc 1m 1s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 16s the patch passed +1 compile 1m 19s the patch passed +1 cc 1m 19s the patch passed +1 javac 1m 19s the patch passed +1 checkstyle 0m 36s the patch passed +1 mvnsite 1m 19s the patch passed +1 mvneclipse 0m 21s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 19s the patch passed +1 javadoc 0m 55s the patch passed +1 unit 0m 55s hadoop-hdfs-client in the patch passed. -1 unit 90m 13s hadoop-hdfs in the patch failed. -1 asflicense 0m 18s The patch generated 1 ASF License warnings. 123m 6s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040 Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11417 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12854780/HDFS-11417.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux de6375b69899 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 05391c1 Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18449/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18449/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/18449/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18449/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          v2 : Added ASF license for DatanodeStorageLocalInfo

          Show
          surendrasingh Surendra Singh Lilhore added a comment - v2 : Added ASF license for DatanodeStorageLocalInfo
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 8s Maven dependency ordering for branch
          +1 mvninstall 14m 51s trunk passed
          +1 compile 1m 36s trunk passed
          +1 checkstyle 0m 43s trunk passed
          +1 mvnsite 1m 41s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          +1 findbugs 3m 52s trunk passed
          +1 javadoc 1m 11s trunk passed
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 1m 39s the patch passed
          +1 compile 1m 42s the patch passed
          +1 cc 1m 42s the patch passed
          +1 javac 1m 42s the patch passed
          +1 checkstyle 0m 42s the patch passed
          +1 mvnsite 1m 35s the patch passed
          +1 mvneclipse 0m 22s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 35s the patch passed
          +1 javadoc 0m 56s the patch passed
          +1 unit 1m 2s hadoop-hdfs-client in the patch passed.
          -1 unit 91m 27s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 41s The patch does not generate ASF License warnings.
          130m 15s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestErasureCodeBenchmarkThroughput
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11417
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12854938/HDFS-11417.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 217a81f8f64f 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11 16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 5f5b031
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18455/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18455/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18455/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 8s Maven dependency ordering for branch +1 mvninstall 14m 51s trunk passed +1 compile 1m 36s trunk passed +1 checkstyle 0m 43s trunk passed +1 mvnsite 1m 41s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 3m 52s trunk passed +1 javadoc 1m 11s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 39s the patch passed +1 compile 1m 42s the patch passed +1 cc 1m 42s the patch passed +1 javac 1m 42s the patch passed +1 checkstyle 0m 42s the patch passed +1 mvnsite 1m 35s the patch passed +1 mvneclipse 0m 22s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 35s the patch passed +1 javadoc 0m 56s the patch passed +1 unit 1m 2s hadoop-hdfs-client in the patch passed. -1 unit 91m 27s hadoop-hdfs in the patch failed. +1 asflicense 0m 41s The patch does not generate ASF License warnings. 130m 15s Reason Tests Failed junit tests hadoop.hdfs.TestErasureCodeBenchmarkThroughput   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11417 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12854938/HDFS-11417.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 217a81f8f64f 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11 16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5f5b031 Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18455/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18455/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18455/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Thanks Surendra Singh Lilhore for the command. It would be a nice addition for admins.

          following are some of my comments.

          1. To make the naming in sync with Datanode UI, instead of "storage", "volume" could be used. Like below,
          "getStorageReport" --> "getVolumeReport()" to NOT to confuse with namenode's report.
          "DatanodeStorageLocalInfo" --> "DatanodeVolumeInfo" to NOT to confuse with existing DatanodeStorageInfo.

          2. Datanode#getStorageReport(..) should check for checkSuperuserPrivilege().

          3. PBHelperClient.convertStorageType(..) could be used instead of {{StorageType.valueOf(proto.getStorageType().toString())}, and vice versa.

          4. name could be just datanode.
          ClientDatanodeProtocol refreshProtocol = getDataNodeProxy(datanode);

          5. Active Storage(3): could be like Active Storages : 3

          6. Write to System.out instead of System.err

          	    System.err.println("Active Storage(" + storageReport.size() + "):");
          	    for (DatanodeStorageLocalInfo info : storageReport) {
          	      System.err.println("\n" + info.getDatanodeStorageLocalReport());
          	    }
          Show
          vinayrpet Vinayakumar B added a comment - Thanks Surendra Singh Lilhore for the command. It would be a nice addition for admins. following are some of my comments. 1. To make the naming in sync with Datanode UI, instead of "storage", "volume" could be used. Like below, "getStorageReport" --> "getVolumeReport()" to NOT to confuse with namenode's report. "DatanodeStorageLocalInfo" --> "DatanodeVolumeInfo" to NOT to confuse with existing DatanodeStorageInfo. 2. Datanode#getStorageReport(..) should check for checkSuperuserPrivilege() . 3. PBHelperClient.convertStorageType(..) could be used instead of {{StorageType.valueOf(proto.getStorageType().toString())}, and vice versa. 4. name could be just datanode . ClientDatanodeProtocol refreshProtocol = getDataNodeProxy(datanode); 5. Active Storage(3): could be like Active Storages : 3 6. Write to System.out instead of System.err System .err.println( "Active Storage(" + storageReport.size() + "):" ); for (DatanodeStorageLocalInfo info : storageReport) { System .err.println( "\n" + info.getDatanodeStorageLocalReport()); }
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Thanks Vinayakumar B for review..
          Attached updated patch, please review..

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Thanks Vinayakumar B for review.. Attached updated patch, please review..
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 7s Maven dependency ordering for branch
          +1 mvninstall 12m 36s trunk passed
          +1 compile 1m 22s trunk passed
          +1 checkstyle 0m 42s trunk passed
          +1 mvnsite 1m 23s trunk passed
          +1 mvneclipse 0m 24s trunk passed
          +1 findbugs 3m 4s trunk passed
          +1 javadoc 1m 1s trunk passed
          0 mvndep 0m 6s Maven dependency ordering for patch
          +1 mvninstall 1m 15s the patch passed
          +1 compile 1m 19s the patch passed
          +1 cc 1m 19s the patch passed
          +1 javac 1m 19s the patch passed
          -0 checkstyle 0m 41s hadoop-hdfs-project: The patch generated 16 new + 357 unchanged - 3 fixed = 373 total (was 360)
          +1 mvnsite 1m 18s the patch passed
          +1 mvneclipse 0m 21s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 18s the patch passed
          +1 javadoc 0m 54s the patch passed
          +1 unit 0m 54s hadoop-hdfs-client in the patch passed.
          -1 unit 64m 52s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          97m 35s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11417
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12855552/HDFS-11417.003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 47732487ea74 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 4e14ead
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18510/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18510/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18510/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18510/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 7s Maven dependency ordering for branch +1 mvninstall 12m 36s trunk passed +1 compile 1m 22s trunk passed +1 checkstyle 0m 42s trunk passed +1 mvnsite 1m 23s trunk passed +1 mvneclipse 0m 24s trunk passed +1 findbugs 3m 4s trunk passed +1 javadoc 1m 1s trunk passed 0 mvndep 0m 6s Maven dependency ordering for patch +1 mvninstall 1m 15s the patch passed +1 compile 1m 19s the patch passed +1 cc 1m 19s the patch passed +1 javac 1m 19s the patch passed -0 checkstyle 0m 41s hadoop-hdfs-project: The patch generated 16 new + 357 unchanged - 3 fixed = 373 total (was 360) +1 mvnsite 1m 18s the patch passed +1 mvneclipse 0m 21s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 18s the patch passed +1 javadoc 0m 54s the patch passed +1 unit 0m 54s hadoop-hdfs-client in the patch passed. -1 unit 64m 52s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 97m 35s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11417 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12855552/HDFS-11417.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 47732487ea74 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4e14ead Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18510/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18510/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18510/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18510/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Fixed checkstyle issue..
          Attached v4 patch.

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Fixed checkstyle issue.. Attached v4 patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 32s Maven dependency ordering for branch
          +1 mvninstall 17m 3s trunk passed
          +1 compile 2m 5s trunk passed
          +1 checkstyle 1m 3s trunk passed
          +1 mvnsite 1m 57s trunk passed
          +1 mvneclipse 0m 34s trunk passed
          +1 findbugs 4m 3s trunk passed
          +1 javadoc 1m 18s trunk passed
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 1m 48s the patch passed
          +1 compile 1m 52s the patch passed
          +1 cc 1m 52s the patch passed
          +1 javac 1m 52s the patch passed
          -0 checkstyle 0m 52s hadoop-hdfs-project: The patch generated 3 new + 357 unchanged - 3 fixed = 360 total (was 360)
          +1 mvnsite 1m 44s the patch passed
          +1 mvneclipse 0m 26s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 4m 14s the patch passed
          +1 javadoc 1m 11s the patch passed
          +1 unit 1m 8s hadoop-hdfs-client in the patch passed.
          +1 unit 92m 44s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          137m 14s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11417
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12855666/HDFS-11417.004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 5aba8c3759ef 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / eeca8b0
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18514/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18514/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18514/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 32s Maven dependency ordering for branch +1 mvninstall 17m 3s trunk passed +1 compile 2m 5s trunk passed +1 checkstyle 1m 3s trunk passed +1 mvnsite 1m 57s trunk passed +1 mvneclipse 0m 34s trunk passed +1 findbugs 4m 3s trunk passed +1 javadoc 1m 18s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 48s the patch passed +1 compile 1m 52s the patch passed +1 cc 1m 52s the patch passed +1 javac 1m 52s the patch passed -0 checkstyle 0m 52s hadoop-hdfs-project: The patch generated 3 new + 357 unchanged - 3 fixed = 360 total (was 360) +1 mvnsite 1m 44s the patch passed +1 mvneclipse 0m 26s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 14s the patch passed +1 javadoc 1m 11s the patch passed +1 unit 1m 8s hadoop-hdfs-client in the patch passed. +1 unit 92m 44s hadoop-hdfs in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 137m 14s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11417 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12855666/HDFS-11417.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 5aba8c3759ef 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / eeca8b0 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18514/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18514/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18514/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Attached v5 patch.
          Fixed checkstyle issues, Please review...

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Attached v5 patch. Fixed checkstyle issues, Please review...
          Hide
          ajisakaa Akira Ajisaka added a comment -
          DataNode.java
                @SuppressWarnings("unchecked")
                Map<String, Object> volumeInfo = (Map<String, Object>) volume.getValue();
          

          This type casting can fail if the existing implementation in FsDatasetImpl.java has been changed. Can we change FsDatasetSpi#getVolumeInfoMap() to return Map<String, Map<String, Object>> to avoid relying on the type casting? What do you think, Vinayakumar B and Surendra Singh Lilhore?

          Show
          ajisakaa Akira Ajisaka added a comment - DataNode.java @SuppressWarnings( "unchecked" ) Map< String , Object > volumeInfo = (Map< String , Object >) volume.getValue(); This type casting can fail if the existing implementation in FsDatasetImpl.java has been changed. Can we change FsDatasetSpi#getVolumeInfoMap() to return Map<String, Map<String, Object>> to avoid relying on the type casting? What do you think, Vinayakumar B and Surendra Singh Lilhore ?
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Thanks Akira Ajisaka for review...

          +1 from my side for this suggestion. We will wait for Vinayakumar B opinion.

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Thanks Akira Ajisaka for review... +1 from my side for this suggestion. We will wait for Vinayakumar B opinion.
          Hide
          vinayrpet Vinayakumar B added a comment -

          This type casting can fail if the existing implementation in FsDatasetImpl.java has been changed

          More errors could come if another implementation is completely different and have different params than expected as below

          +    for (Entry<String, Object> volume : volumeInfoMap.entrySet()) {
          +      @SuppressWarnings("unchecked")
          +      Map<String, Object> volumeInfo = (Map<String, Object>) volume.getValue();
          +      DatanodeVolumeInfo dnStorageInfo = new DatanodeVolumeInfo(
          +          volume.getKey(), (Long) volumeInfo.get("usedSpace"),
          +          (Long) volumeInfo.get("freeSpace"),
          +          (Long) volumeInfo.get("reservedSpace"),
          +          (Long) volumeInfo.get("reservedSpaceForReplicas"),
          +          (Long) volumeInfo.get("numBlocks"),
          +          (StorageType) volumeInfo.get("storageType"));
          +      volumeInfoList.add(dnStorageInfo);
          +    }

          I feel current API signature is better, as getVolumeInfoMap() implementation could be completely different for different implementation.
          IMO, In current Jira, it would be sufficient to handle existing implementations (FSDataSetImpl, ExternalDatasetImpl (which returns null, so null check could be added)).
          If anything more required, could be handled in follow-up Jiras.

          Show
          vinayrpet Vinayakumar B added a comment - This type casting can fail if the existing implementation in FsDatasetImpl.java has been changed More errors could come if another implementation is completely different and have different params than expected as below + for (Entry< String , Object > volume : volumeInfoMap.entrySet()) { + @SuppressWarnings( "unchecked" ) + Map< String , Object > volumeInfo = (Map< String , Object >) volume.getValue(); + DatanodeVolumeInfo dnStorageInfo = new DatanodeVolumeInfo( + volume.getKey(), ( Long ) volumeInfo.get( "usedSpace" ), + ( Long ) volumeInfo.get( "freeSpace" ), + ( Long ) volumeInfo.get( "reservedSpace" ), + ( Long ) volumeInfo.get( "reservedSpaceForReplicas" ), + ( Long ) volumeInfo.get( "numBlocks" ), + (StorageType) volumeInfo.get( "storageType" )); + volumeInfoList.add(dnStorageInfo); + } I feel current API signature is better, as getVolumeInfoMap() implementation could be completely different for different implementation. IMO, In current Jira, it would be sufficient to handle existing implementations (FSDataSetImpl, ExternalDatasetImpl (which returns null, so null check could be added)). If anything more required, could be handled in follow-up Jiras.
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Thanks Vinayakumar B.
          Akira Ajisaka are you agree with Vinay's opinion ?

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Thanks Vinayakumar B . Akira Ajisaka are you agree with Vinay's opinion ?
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Yes, agreed.

          Show
          ajisakaa Akira Ajisaka added a comment - Yes, agreed.
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Thanks Vinayakumar B and Akira Ajisaka.
          Attached v6 patch. Added null check for volumeInfoMap.
          Please review..

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Thanks Vinayakumar B and Akira Ajisaka . Attached v6 patch. Added null check for volumeInfoMap . Please review..
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 7s Maven dependency ordering for branch
          +1 mvninstall 13m 16s trunk passed
          +1 compile 1m 24s trunk passed
          +1 checkstyle 0m 45s trunk passed
          +1 mvnsite 1m 25s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          -1 findbugs 1m 18s hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings.
          -1 findbugs 1m 39s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
          +1 javadoc 1m 1s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 18s the patch passed
          +1 compile 1m 22s the patch passed
          +1 cc 1m 22s the patch passed
          +1 javac 1m 22s the patch passed
          -0 checkstyle 0m 42s hadoop-hdfs-project: The patch generated 2 new + 356 unchanged - 3 fixed = 358 total (was 359)
          +1 mvnsite 1m 22s the patch passed
          +1 mvneclipse 0m 23s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 9s the patch passed
          +1 javadoc 0m 55s the patch passed
          +1 unit 1m 9s hadoop-hdfs-client in the patch passed.
          -1 unit 63m 3s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          96m 55s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.balancer.TestBalancer



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ac17dc
          JIRA Issue HDFS-11417
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12864334/HDFS-11417.006.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 9739ad63c70c 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c0cf11e
          Default Java 1.8.0_121
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19162/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19162/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19162/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19162/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19162/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19162/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 7s Maven dependency ordering for branch +1 mvninstall 13m 16s trunk passed +1 compile 1m 24s trunk passed +1 checkstyle 0m 45s trunk passed +1 mvnsite 1m 25s trunk passed +1 mvneclipse 0m 28s trunk passed -1 findbugs 1m 18s hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings. -1 findbugs 1m 39s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 1m 1s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 18s the patch passed +1 compile 1m 22s the patch passed +1 cc 1m 22s the patch passed +1 javac 1m 22s the patch passed -0 checkstyle 0m 42s hadoop-hdfs-project: The patch generated 2 new + 356 unchanged - 3 fixed = 358 total (was 359) +1 mvnsite 1m 22s the patch passed +1 mvneclipse 0m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 9s the patch passed +1 javadoc 0m 55s the patch passed +1 unit 1m 9s hadoop-hdfs-client in the patch passed. -1 unit 63m 3s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 96m 55s Reason Tests Failed junit tests hadoop.hdfs.server.balancer.TestBalancer Subsystem Report/Notes Docker Image:yetus/hadoop:0ac17dc JIRA Issue HDFS-11417 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12864334/HDFS-11417.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 9739ad63c70c 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c0cf11e Default Java 1.8.0_121 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19162/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19162/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19162/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19162/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19162/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19162/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Findbugs warnings are not related to this jira..
          I feel we no need to handle checkstyle warning .

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Findbugs warnings are not related to this jira.. I feel we no need to handle checkstyle warning .
          Hide
          ajisakaa Akira Ajisaka added a comment -

          LGTM, +1

          Show
          ajisakaa Akira Ajisaka added a comment - LGTM, +1
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Thanks Akira Ajisaka for review. Can you commit this ?

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Thanks Akira Ajisaka for review. Can you commit this ?
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Committed this to trunk and branch-2. Thanks Surendra Singh Lilhore for the contribution and thanks Vinayakumar B for the review!

          Show
          ajisakaa Akira Ajisaka added a comment - Committed this to trunk and branch-2. Thanks Surendra Singh Lilhore for the contribution and thanks Vinayakumar B for the review!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11634 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11634/)
          HDFS-11417. Add datanode admin command to get the storage info. (aajisaka: rev 93fa48fcf243dc759db1736af145633da760f937)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSAdmin.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
          • (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeVolumeInfo.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolTranslatorPB.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolServerSideTranslatorPB.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/ClientDatanodeProtocol.proto
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11634 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11634/ ) HDFS-11417 . Add datanode admin command to get the storage info. (aajisaka: rev 93fa48fcf243dc759db1736af145633da760f937) (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSAdmin.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeVolumeInfo.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolTranslatorPB.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolServerSideTranslatorPB.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/ClientDatanodeProtocol.proto (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Thanks Akira Ajisaka for review and commit. Thanks Vinayakumar B for review..

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Thanks Akira Ajisaka for review and commit. Thanks Vinayakumar B for review..

            People

            • Assignee:
              surendrasingh Surendra Singh Lilhore
              Reporter:
              surendrasingh Surendra Singh Lilhore
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development