Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12070

Some of the bin/hadoop subcommands are not available on Windows

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: scripts
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      • conftest, distch, jnipath and trace are not enabled in hadoop.cmd
      • kerbname is enabled, but does not appear in the help message

        Activity

        Hide
        sekikn Kengo Seki added a comment -

        Test results on my environment. It seems to work.

        • conftest
        C:\hadoop-3.0.0-SNAPSHOT>bin\hadoop conftest
        C:\hadoop-3.0.0-SNAPSHOT\etc\hadoop\capacity-scheduler.xml: valid
        C:\hadoop-3.0.0-SNAPSHOT\etc\hadoop\core-site.xml: valid
        C:\hadoop-3.0.0-SNAPSHOT\etc\hadoop\hadoop-policy.xml: valid
        C:\hadoop-3.0.0-SNAPSHOT\etc\hadoop\hdfs-site.xml: valid
        C:\hadoop-3.0.0-SNAPSHOT\etc\hadoop\httpfs-site.xml: valid
        C:\hadoop-3.0.0-SNAPSHOT\etc\hadoop\kms-acls.xml: valid
        C:\hadoop-3.0.0-SNAPSHOT\etc\hadoop\kms-site.xml: valid
        C:\hadoop-3.0.0-SNAPSHOT\etc\hadoop\mapred-site.xml: valid
        C:\hadoop-3.0.0-SNAPSHOT\etc\hadoop\yarn-site.xml: valid
        OK
        
        • distch
        C:\hadoop-3.0.0-SNAPSHOT>bin\hdfs dfs -ls /tmp/a
        -rw-r--r--   1 sekikn supergroup          0 2015-06-07 00:15 /tmp/a
        
        C:\hadoop-3.0.0-SNAPSHOT>bin\hadoop distch /tmp/a:::666
        15/06/07 00:16:06 INFO tools.DistTool: ops=[/tmp/a:null:null:rw-rw-rw-]
        15/06/07 00:16:06 INFO tools.DistTool: isIgnoreFailures=false
        15/06/07 00:16:09 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
        15/06/07 00:16:09 INFO tools.DistTool: distch.job.dir=/tmp/hadoop-yarn/staging/sekikn/.stagingdistch_qh3wa4
        15/06/07 00:16:09 INFO tools.DistTool: log=/tmp/hadoop-yarn/staging/sekikn/.stagingdistch_qh3wa4/_logs
        15/06/07 00:16:09 INFO Configuration.deprecation: io.sort.mb is deprecated. Instead, use mapreduce.task.io.sort.mb
        15/06/07 00:16:09 INFO Configuration.deprecation: io.sort.factor is deprecated. Instead, use mapreduce.task.io.sort.factor
        15/06/07 00:16:10 INFO tools.DistTool: distch.op.count=1
        15/06/07 00:16:10 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
        15/06/07 00:16:10 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
        15/06/07 00:16:10 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
        15/06/07 00:16:11 INFO tools.DistTool: numSplits=1, splits.size()=1
        15/06/07 00:16:11 INFO mapreduce.JobSubmitter: number of splits:1
        15/06/07 00:16:12 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1433603668279_0002
        15/06/07 00:16:12 INFO impl.YarnClientImpl: Submitted application application_1433603668279_0002
        15/06/07 00:16:12 INFO mapreduce.Job: The url to track the job: http://sekikn-PC:8088/proxy/application_1433603668279_0002/
        15/06/07 00:16:12 INFO mapreduce.Job: Running job: job_1433603668279_0002
        15/06/07 00:16:22 INFO mapreduce.Job: Job job_1433603668279_0002 running in uber mode : false
        15/06/07 00:16:22 INFO mapreduce.Job:  map 0% reduce 0%
        15/06/07 00:16:29 INFO mapreduce.Job:  map 100% reduce 0%
        15/06/07 00:16:31 INFO mapreduce.Job: Job job_1433603668279_0002 completed successfully
        15/06/07 00:16:31 INFO mapreduce.Job: Counters: 31
                File System Counters
                        FILE: Number of bytes read=0
                        FILE: Number of bytes written=111250
                        FILE: Number of read operations=0
                        FILE: Number of large read operations=0
                        FILE: Number of write operations=0
                        HDFS: Number of bytes read=283
                        HDFS: Number of bytes written=0
                        HDFS: Number of read operations=8
                        HDFS: Number of large read operations=0
                        HDFS: Number of write operations=3
                Job Counters
                        Launched map tasks=1
                        Other local map tasks=1
                        Total time spent by all maps in occupied slots (ms)=5512
                        Total time spent by all reduces in occupied slots (ms)=0
                        Total time spent by all map tasks (ms)=5512
                        Total vcore-seconds taken by all map tasks=5512
                        Total megabyte-seconds taken by all map tasks=5644288
                Map-Reduce Framework
                        Map input records=1
                        Map output records=0
                        Input split bytes=121
                        Spilled Records=0
                        Failed Shuffles=0
                        Merged Map outputs=0
                        GC time elapsed (ms)=34
                        CPU time spent (ms)=577
                        Physical memory (bytes) snapshot=181915648
                        Virtual memory (bytes) snapshot=332591104
                        Total committed heap usage (bytes)=114884608
                File Input Format Counters
                        Bytes Read=162
                File Output Format Counters
                        Bytes Written=0
                org.apache.hadoop.tools.DistCh$Counter
                        SUCCEED=1
        
        C:\hadoop-3.0.0-SNAPSHOT>bin\hdfs dfs -ls /tmp/a
        -rw-rw-rw-   1 sekikn supergroup          0 2015-06-07 00:15 /tmp/a
        
        • jnipath
        C:\hadoop-3.0.0-SNAPSHOT>bin\hadoop jnipath
        C:\hadoop-3.0.0-SNAPSHOT\bin;C:\Windows\Microsoft.NET\Framework64\v4.0.30319;C:\Windows\Microsoft.NET\Framework\v4.0.30319;C:\Windows\Microsoft.NET\Framework64\v3.5;C:\Windows\Microsoft.NET\Framework\v3.5; (snip)
        
        • trace
        C:\hadoop-3.0.0-SNAPSHOT>bin\hadoop trace -list -host 127.0.0.1:50020
        [no span receivers found]
        
        C:\hadoop-3.0.0-SNAPSHOT>bin\hadoop trace -add -host 127.0.0.1:50020 -class StandardOutSpanReceiver
        Added trace span receiver 1 with configuration
        
        C:\hadoop-3.0.0-SNAPSHOT>bin\hadoop trace -list -host 127.0.0.1:50020
        ID  CLASS
        1   org.apache.htrace.impl.StandardOutSpanReceiver
        
        • kerbname (in the help message)
        C:\hadoop-3.0.0-SNAPSHOT>bin\hadoop
        Usage: hadoop [--config confdir] [--loglevel loglevel] COMMAND
        where COMMAND is one of:
          fs                   run a generic filesystem user client
          version              print the version
          jar <jar>            run a jar file
                               note: please use "yarn jar" to launch
                                     YARN applications, not this command.
          checknative [-a|-h]  check native hadoop and compression libraries availability
          conftest             validate configuration XML files
          distch path:owner:group:permisson
                               distributed metadata changer
          distcp <srcurl> <desturl> copy file or directories recursively
          archive -archiveName NAME -p <parent path> <src>* <dest> create a hadoop archive
          classpath            prints the class path needed to get the
                               Hadoop jar and the required libraries
          credential           interact with credential providers
          jnipath              prints the java.library.path
          kerbname             show auth_to_local principal conversion
          key                  manage keys via the KeyProvider
          trace                view and modify Hadoop tracing settings
          daemonlog            get/set the log level for each daemon
         or
          CLASSNAME            run the class named CLASSNAME
        
        Most commands print help when invoked w/o parameters.
        
        Show
        sekikn Kengo Seki added a comment - Test results on my environment. It seems to work. conftest C:\hadoop-3.0.0-SNAPSHOT>bin\hadoop conftest C:\hadoop-3.0.0-SNAPSHOT\etc\hadoop\capacity-scheduler.xml: valid C:\hadoop-3.0.0-SNAPSHOT\etc\hadoop\core-site.xml: valid C:\hadoop-3.0.0-SNAPSHOT\etc\hadoop\hadoop-policy.xml: valid C:\hadoop-3.0.0-SNAPSHOT\etc\hadoop\hdfs-site.xml: valid C:\hadoop-3.0.0-SNAPSHOT\etc\hadoop\httpfs-site.xml: valid C:\hadoop-3.0.0-SNAPSHOT\etc\hadoop\kms-acls.xml: valid C:\hadoop-3.0.0-SNAPSHOT\etc\hadoop\kms-site.xml: valid C:\hadoop-3.0.0-SNAPSHOT\etc\hadoop\mapred-site.xml: valid C:\hadoop-3.0.0-SNAPSHOT\etc\hadoop\yarn-site.xml: valid OK distch C:\hadoop-3.0.0-SNAPSHOT>bin\hdfs dfs -ls /tmp/a -rw-r--r-- 1 sekikn supergroup 0 2015-06-07 00:15 /tmp/a C:\hadoop-3.0.0-SNAPSHOT>bin\hadoop distch /tmp/a:::666 15/06/07 00:16:06 INFO tools.DistTool: ops=[/tmp/a: null : null :rw-rw-rw-] 15/06/07 00:16:06 INFO tools.DistTool: isIgnoreFailures= false 15/06/07 00:16:09 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 15/06/07 00:16:09 INFO tools.DistTool: distch.job.dir=/tmp/hadoop-yarn/staging/sekikn/.stagingdistch_qh3wa4 15/06/07 00:16:09 INFO tools.DistTool: log=/tmp/hadoop-yarn/staging/sekikn/.stagingdistch_qh3wa4/_logs 15/06/07 00:16:09 INFO Configuration.deprecation: io.sort.mb is deprecated. Instead, use mapreduce.task.io.sort.mb 15/06/07 00:16:09 INFO Configuration.deprecation: io.sort.factor is deprecated. Instead, use mapreduce.task.io.sort.factor 15/06/07 00:16:10 INFO tools.DistTool: distch.op.count=1 15/06/07 00:16:10 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 15/06/07 00:16:10 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 15/06/07 00:16:10 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 15/06/07 00:16:11 INFO tools.DistTool: numSplits=1, splits.size()=1 15/06/07 00:16:11 INFO mapreduce.JobSubmitter: number of splits:1 15/06/07 00:16:12 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1433603668279_0002 15/06/07 00:16:12 INFO impl.YarnClientImpl: Submitted application application_1433603668279_0002 15/06/07 00:16:12 INFO mapreduce.Job: The url to track the job: http: //sekikn-PC:8088/proxy/application_1433603668279_0002/ 15/06/07 00:16:12 INFO mapreduce.Job: Running job: job_1433603668279_0002 15/06/07 00:16:22 INFO mapreduce.Job: Job job_1433603668279_0002 running in uber mode : false 15/06/07 00:16:22 INFO mapreduce.Job: map 0% reduce 0% 15/06/07 00:16:29 INFO mapreduce.Job: map 100% reduce 0% 15/06/07 00:16:31 INFO mapreduce.Job: Job job_1433603668279_0002 completed successfully 15/06/07 00:16:31 INFO mapreduce.Job: Counters: 31 File System Counters FILE: Number of bytes read=0 FILE: Number of bytes written=111250 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=283 HDFS: Number of bytes written=0 HDFS: Number of read operations=8 HDFS: Number of large read operations=0 HDFS: Number of write operations=3 Job Counters Launched map tasks=1 Other local map tasks=1 Total time spent by all maps in occupied slots (ms)=5512 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=5512 Total vcore-seconds taken by all map tasks=5512 Total megabyte-seconds taken by all map tasks=5644288 Map-Reduce Framework Map input records=1 Map output records=0 Input split bytes=121 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=34 CPU time spent (ms)=577 Physical memory (bytes) snapshot=181915648 Virtual memory (bytes) snapshot=332591104 Total committed heap usage (bytes)=114884608 File Input Format Counters Bytes Read=162 File Output Format Counters Bytes Written=0 org.apache.hadoop.tools.DistCh$Counter SUCCEED=1 C:\hadoop-3.0.0-SNAPSHOT>bin\hdfs dfs -ls /tmp/a -rw-rw-rw- 1 sekikn supergroup 0 2015-06-07 00:15 /tmp/a jnipath C:\hadoop-3.0.0-SNAPSHOT>bin\hadoop jnipath C:\hadoop-3.0.0-SNAPSHOT\bin;C:\Windows\Microsoft.NET\Framework64\v4.0.30319;C:\Windows\Microsoft.NET\Framework\v4.0.30319;C:\Windows\Microsoft.NET\Framework64\v3.5;C:\Windows\Microsoft.NET\Framework\v3.5; (snip) trace C:\hadoop-3.0.0-SNAPSHOT>bin\hadoop trace -list -host 127.0.0.1:50020 [no span receivers found] C:\hadoop-3.0.0-SNAPSHOT>bin\hadoop trace -add -host 127.0.0.1:50020 -class StandardOutSpanReceiver Added trace span receiver 1 with configuration C:\hadoop-3.0.0-SNAPSHOT>bin\hadoop trace -list -host 127.0.0.1:50020 ID CLASS 1 org.apache.htrace.impl.StandardOutSpanReceiver kerbname (in the help message) C:\hadoop-3.0.0-SNAPSHOT>bin\hadoop Usage: hadoop [--config confdir] [--loglevel loglevel] COMMAND where COMMAND is one of: fs run a generic filesystem user client version print the version jar <jar> run a jar file note: please use "yarn jar" to launch YARN applications, not this command. checknative [-a|-h] check native hadoop and compression libraries availability conftest validate configuration XML files distch path:owner:group:permisson distributed metadata changer distcp <srcurl> <desturl> copy file or directories recursively archive -archiveName NAME -p <parent path> <src>* <dest> create a hadoop archive classpath prints the class path needed to get the Hadoop jar and the required libraries credential interact with credential providers jnipath prints the java.library.path kerbname show auth_to_local principal conversion key manage keys via the KeyProvider trace view and modify Hadoop tracing settings daemonlog get/set the log level for each daemon or CLASSNAME run the class named CLASSNAME Most commands print help when invoked w/o parameters.
        Hide
        hadoopqa Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 0m 0s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 release audit 0m 14s The applied patch does not increase the total number of release audit warnings.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
            0m 18s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12738174/HADOOP-12070.001.patch
        Optional Tests  
        git revision trunk / e46cb80
        Java 1.7.0_55
        uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6933/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 0m 0s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 release audit 0m 14s The applied patch does not increase the total number of release audit warnings. +1 whitespace 0m 0s The patch has no lines that end in whitespace.     0m 18s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12738174/HADOOP-12070.001.patch Optional Tests   git revision trunk / e46cb80 Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6933/console This message was automatically generated.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        +1 for the patch. I will commit this to trunk today. Thanks for the contribution Kengo Seki.

        For the trace and distch commands I just tested that the commands can be launched. Verified the behavior of the remaining commands.

        xmllint is not available on branch-2 so you can either backport HADOOP-7497 or if you post a branch-2 patch I will commit that too.

        Show
        arpitagarwal Arpit Agarwal added a comment - +1 for the patch. I will commit this to trunk today. Thanks for the contribution Kengo Seki . For the trace and distch commands I just tested that the commands can be launched. Verified the behavior of the remaining commands. xmllint is not available on branch-2 so you can either backport HADOOP-7497 or if you post a branch-2 patch I will commit that too.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        Pushed to trunk.

        Show
        arpitagarwal Arpit Agarwal added a comment - Pushed to trunk.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9362 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9362/)
        HADOOP-12070. Some of the bin/hadoop subcommands are not available on (arp: rev 2e76c2f751f697ed2f785038c22445251db0134c)

        • hadoop-common-project/hadoop-common/src/main/bin/hadoop.cmd
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9362 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9362/ ) HADOOP-12070 . Some of the bin/hadoop subcommands are not available on (arp: rev 2e76c2f751f697ed2f785038c22445251db0134c) hadoop-common-project/hadoop-common/src/main/bin/hadoop.cmd hadoop-common-project/hadoop-common/CHANGES.txt

          People

          • Assignee:
            sekikn Kengo Seki
            Reporter:
            sekikn Kengo Seki
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development