Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-6831

Inconsistency between 'hdfs dfsadmin' and 'hdfs dfsadmin -help'

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.6.0
    • Component/s: None
    • Labels:
    • Hadoop Flags:
      Reviewed

      Description

      There is an inconsistency between the console outputs of 'hdfs dfsadmin' command and 'hdfs dfsadmin -help' command.

      [root@trunk ~]# hdfs dfsadmin
      Usage: java DFSAdmin
      Note: Administrative commands can only be run as the HDFS superuser.
                 [-report]
                 [-safemode enter | leave | get | wait]
                 [-allowSnapshot <snapshotDir>]
                 [-disallowSnapshot <snapshotDir>]
                 [-saveNamespace]
                 [-rollEdits]
                 [-restoreFailedStorage true|false|check]
                 [-refreshNodes]
                 [-finalizeUpgrade]
                 [-rollingUpgrade [<query|prepare|finalize>]]
                 [-metasave filename]
                 [-refreshServiceAcl]
                 [-refreshUserToGroupsMappings]
                 [-refreshSuperUserGroupsConfiguration]
                 [-refreshCallQueue]
                 [-refresh]
                 [-printTopology]
                 [-refreshNamenodes datanodehost:port]
                 [-deleteBlockPool datanode-host:port blockpoolId [force]]
                 [-setQuota <quota> <dirname>...<dirname>]
                 [-clrQuota <dirname>...<dirname>]
                 [-setSpaceQuota <quota> <dirname>...<dirname>]
                 [-clrSpaceQuota <dirname>...<dirname>]
                 [-setBalancerBandwidth <bandwidth in bytes per second>]
                 [-fetchImage <local directory>]
                 [-shutdownDatanode <datanode_host:ipc_port> [upgrade]]
                 [-getDatanodeInfo <datanode_host:ipc_port>]
                 [-help [cmd]]
      
      [root@trunk ~]# hdfs dfsadmin -help
      hadoop dfsadmin performs DFS administrative commands.
      The full syntax is: 
      
      hadoop dfsadmin
      	[-report [-live] [-dead] [-decommissioning]]
      	[-safemode <enter | leave | get | wait>]
      	[-saveNamespace]
      	[-rollEdits]
      	[-restoreFailedStorage true|false|check]
      	[-refreshNodes]
      	[-setQuota <quota> <dirname>...<dirname>]
      	[-clrQuota <dirname>...<dirname>]
      	[-setSpaceQuota <quota> <dirname>...<dirname>]
      	[-clrSpaceQuota <dirname>...<dirname>]
      	[-finalizeUpgrade]
      	[-rollingUpgrade [<query|prepare|finalize>]]
      	[-refreshServiceAcl]
      	[-refreshUserToGroupsMappings]
      	[-refreshSuperUserGroupsConfiguration]
      	[-refreshCallQueue]
      	[-refresh <host:ipc_port> <key> [arg1..argn]
      	[-printTopology]
      	[-refreshNamenodes datanodehost:port]
      	[-deleteBlockPool datanodehost:port blockpoolId [force]]
      	[-setBalancerBandwidth <bandwidth>]
      	[-fetchImage <local directory>]
      	[-allowSnapshot <snapshotDir>]
      	[-disallowSnapshot <snapshotDir>]
      	[-shutdownDatanode <datanode_host:ipc_port> [upgrade]]
      	[-getDatanodeInfo <datanode_host:ipc_port>
      	[-help [cmd]
      

      These two outputs should be the same.

      1. HDFS-6831.4.patch
        16 kB
        Xiaoyu Yao
      2. HDFS-6831.3.patch
        13 kB
        Xiaoyu Yao
      3. HDFS-6831.2.patch
        16 kB
        Xiaoyu Yao
      4. HDFS-6831.1.patch
        13 kB
        Xiaoyu Yao
      5. HDFS-6831.0.patch
        13 kB
        Xiaoyu Yao

        Activity

        Hide
        Jing Zhao added a comment -

        Also in "dfsadmin -help" there are only a couple of commands' help information mention that it "requires superuser permissions". Maybe we can move this phrase into the help summary.

        Show
        Jing Zhao added a comment - Also in "dfsadmin -help" there are only a couple of commands' help information mention that it "requires superuser permissions". Maybe we can move this phrase into the help summary.
        Hide
        Xiaoyu Yao added a comment -

        Attach a patch that fixed the issue with a commonUsageSummary string for both commands and misc minor fixes.

        • add the missing "-metasave filename"
        • add the "Note: Administrative commands can only be run with superuser permission.\n" to summary of PrintHelp() as Jing suggested.
        • change "java DfsAdmin" to "hdfs DfsAdmin" in PrintUsage()

        Not sure why quota and rolling upgrade commands have different for display format with two "+" instead of "-" and extra spaces. I just keep it as is and guess maybe it is to differentiate as new features.

        "\t[" + SetQuotaCommand.USAGE + "]\n" +

        Show
        Xiaoyu Yao added a comment - Attach a patch that fixed the issue with a commonUsageSummary string for both commands and misc minor fixes. add the missing "-metasave filename" add the "Note: Administrative commands can only be run with superuser permission.\n" to summary of PrintHelp() as Jing suggested. change "java DfsAdmin" to "hdfs DfsAdmin" in PrintUsage() Not sure why quota and rolling upgrade commands have different for display format with two "+" instead of "-" and extra spaces. I just keep it as is and guess maybe it is to differentiate as new features. "\t [" + SetQuotaCommand.USAGE + "] \n" +
        Hide
        Akira AJISAKA added a comment -

        Thanks Nathan Yao for the patch!
        Would you change "hdfs DFSAdmin" to "hdfs dfsadmin"?

        Show
        Akira AJISAKA added a comment - Thanks Nathan Yao for the patch! Would you change "hdfs DFSAdmin" to "hdfs dfsadmin"?
        Hide
        Xiaoyu Yao added a comment -

        Update "hdfs DFSAdmin" to "hdfs dfsadmin"

        Show
        Xiaoyu Yao added a comment - Update "hdfs DFSAdmin" to "hdfs dfsadmin"
        Hide
        Arpit Agarwal added a comment -

        +1 pending Jenkins.

        Show
        Arpit Agarwal added a comment - +1 pending Jenkins.
        Hide
        Akira AJISAKA added a comment -

        One more comment:

              "The full syntax is: \n\n" +
              "hadoop dfsadmin\n" +
              commonUsageSummary;
        

        Would you change "hadoop dfsadmin" to "hdfs dfsadmin"? I'm +1 (non-binding) if that is included.

            String summary = "hadoop dfsadmin performs DFS administrative commands.\n" +
        

        I'm thinking the above can be changed to "hdfs dfsadmin" also. Arpit Agarwal and Xiaoyu Yao, what do you think?

        Show
        Akira AJISAKA added a comment - One more comment: "The full syntax is: \n\n" + "hadoop dfsadmin\n" + commonUsageSummary; Would you change "hadoop dfsadmin" to "hdfs dfsadmin"? I'm +1 (non-binding) if that is included. String summary = "hadoop dfsadmin performs DFS administrative commands.\n" + I'm thinking the above can be changed to "hdfs dfsadmin" also. Arpit Agarwal and Xiaoyu Yao , what do you think?
        Hide
        Arpit Agarwal added a comment -

        Good catch. Xiaoyu Yao can you please update the patch with Akira's suggestions?

        Show
        Arpit Agarwal added a comment - Good catch. Xiaoyu Yao can you please update the patch with Akira's suggestions?
        Hide
        Xiaoyu Yao added a comment -

        Update patch based on Akira AJISAKA and Arpit Agarwal' feedback.

        Show
        Xiaoyu Yao added a comment - Update patch based on Akira AJISAKA and Arpit Agarwal ' feedback.
        Hide
        Akira AJISAKA added a comment -

        Thanks Xiaoyu Yao for updating the patch. Would you please remove grouping the imports from the patch?

        Show
        Akira AJISAKA added a comment - Thanks Xiaoyu Yao for updating the patch. Would you please remove grouping the imports from the patch?
        Hide
        Xiaoyu Yao added a comment -

        Thanks! The IntelliJ Optimize import on the fly and does not give a switch to disable it. I will remove the changes around imports from HDFS-6831.2.patch.

        Show
        Xiaoyu Yao added a comment - Thanks! The IntelliJ Optimize import on the fly and does not give a switch to disable it. I will remove the changes around imports from HDFS-6831 .2.patch.
        Hide
        Xiaoyu Yao added a comment -

        Figure out how to disable the import optimize on the fly option from IntelliJ and regenerate the patch.

        Show
        Xiaoyu Yao added a comment - Figure out how to disable the import optimize on the fly option from IntelliJ and regenerate the patch.
        Hide
        Arpit Agarwal added a comment -

        +1 pending Jenkins. Not sure why the earlier builds didn't show results. This one is running at https://builds.apache.org/job/PreCommit-HDFS-Build/7911/

        Show
        Arpit Agarwal added a comment - +1 pending Jenkins. Not sure why the earlier builds didn't show results. This one is running at https://builds.apache.org/job/PreCommit-HDFS-Build/7911/
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12666650/HDFS-6831.2.patch
        against trunk revision 6104520.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.server.namenode.TestBackupNode
        org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool
        org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
        org.apache.hadoop.hdfs.server.datanode.TestBPOfferService
        org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7906//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7906//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12666650/HDFS-6831.2.patch against trunk revision 6104520. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.TestBackupNode org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract org.apache.hadoop.hdfs.server.datanode.TestBPOfferService org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7906//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7906//console This message is automatically generated.
        Hide
        Akira AJISAKA added a comment -

        +1 (non-binding), pending Jenkins to run tests with the v3 patch.

        Show
        Akira AJISAKA added a comment - +1 (non-binding), pending Jenkins to run tests with the v3 patch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12666668/HDFS-6831.3.patch
        against trunk revision 772d1fb.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.tools.TestTools
        org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
        org.apache.hadoop.hdfs.server.balancer.TestBalancer

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7911//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7911//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12666668/HDFS-6831.3.patch against trunk revision 772d1fb. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.tools.TestTools org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover org.apache.hadoop.hdfs.server.balancer.TestBalancer +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7911//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7911//console This message is automatically generated.
        Hide
        Akira AJISAKA added a comment -

        TestTools fails with v3 patch. Would you fix the test?

            String pattern = "Usage: java DFSAdmin";
            checkOutput(new String[] { "-cancel", "-renew" }, pattern, System.err,
                DFSAdmin.class);
        
        Show
        Akira AJISAKA added a comment - TestTools fails with v3 patch. Would you fix the test? String pattern = "Usage: java DFSAdmin" ; checkOutput( new String [] { "-cancel" , "-renew" }, pattern, System .err, DFSAdmin.class);
        Hide
        Xiaoyu Yao added a comment -

        Thanks Akira AJISAKA, patch updated.

        Show
        Xiaoyu Yao added a comment - Thanks Akira AJISAKA , patch updated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12666801/HDFS-6831.4.patch
        against trunk revision 45efc96.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover

        The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.TestDFSInotifyEventInputStream

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7914//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7914//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12666801/HDFS-6831.4.patch against trunk revision 45efc96. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestDFSInotifyEventInputStream +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7914//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7914//console This message is automatically generated.
        Hide
        Xiaoyu Yao added a comment -

        I tired both tests on my machine twice and passed.
        org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
        org.apache.hadoop.hdfs.TestDFSInotifyEventInputStream

        Show
        Xiaoyu Yao added a comment - I tired both tests on my machine twice and passed. org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover org.apache.hadoop.hdfs.TestDFSInotifyEventInputStream
        Hide
        Arpit Agarwal added a comment -

        +1 I committed this to trunk and branch-2.

        Thanks for the contribution Xiaoyu Yao and thanks Akira AJISAKA for reviewing.

        Show
        Arpit Agarwal added a comment - +1 I committed this to trunk and branch-2. Thanks for the contribution Xiaoyu Yao and thanks Akira AJISAKA for reviewing.
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #672 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/672/)
        HDFS-6831. Inconsistency between 'hdfs dfsadmin' and 'hdfs dfsadmin -help'. (Contributed by Xiaoyu Yao) (arp: rev 9e941d9f99168cae01f8d50622a616fc26c196d9)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestTools.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #672 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/672/ ) HDFS-6831 . Inconsistency between 'hdfs dfsadmin' and 'hdfs dfsadmin -help'. (Contributed by Xiaoyu Yao) (arp: rev 9e941d9f99168cae01f8d50622a616fc26c196d9) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestTools.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #1888 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1888/)
        HDFS-6831. Inconsistency between 'hdfs dfsadmin' and 'hdfs dfsadmin -help'. (Contributed by Xiaoyu Yao) (arp: rev 9e941d9f99168cae01f8d50622a616fc26c196d9)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestTools.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1888 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1888/ ) HDFS-6831 . Inconsistency between 'hdfs dfsadmin' and 'hdfs dfsadmin -help'. (Contributed by Xiaoyu Yao) (arp: rev 9e941d9f99168cae01f8d50622a616fc26c196d9) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestTools.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk #1863 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1863/)
        HDFS-6831. Inconsistency between 'hdfs dfsadmin' and 'hdfs dfsadmin -help'. (Contributed by Xiaoyu Yao) (arp: rev 9e941d9f99168cae01f8d50622a616fc26c196d9)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestTools.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1863 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1863/ ) HDFS-6831 . Inconsistency between 'hdfs dfsadmin' and 'hdfs dfsadmin -help'. (Contributed by Xiaoyu Yao) (arp: rev 9e941d9f99168cae01f8d50622a616fc26c196d9) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestTools.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

          People

          • Assignee:
            Xiaoyu Yao
            Reporter:
            Akira AJISAKA
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development