Hadoop Common
  1. Hadoop Common
  2. HADOOP-8431

Running distcp wo args throws IllegalArgumentException

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: None
    • Labels:
    • Hadoop Flags:
      Reviewed

      Description

      Running distcp w/o args results in the following:

      hadoop-3.0.0-SNAPSHOT $ ./bin/hadoop distcp
      12/05/23 18:49:04 ERROR tools.DistCp: Invalid arguments: 
      java.lang.IllegalArgumentException: Target path not specified
      	at org.apache.hadoop.tools.OptionsParser.parse(OptionsParser.java:86)
      	at org.apache.hadoop.tools.DistCp.run(DistCp.java:102)
      	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
      	at org.apache.hadoop.tools.DistCp.main(DistCp.java:368)
      Invalid arguments: Target path not specified
      
      1. diff1.txt
        0.9 kB
        Sandy Ryza

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        24d 6h 20m 1 Sho Shimauchi 17/Jun/12 09:11
        Resolved Resolved Reopened Reopened
        36d 18h 11m 1 Eli Collins 24/Jul/12 03:22
        Reopened Reopened Patch Available Patch Available
        44d 15h 44m 1 Sandy Ryza 06/Sep/12 19:07
        Patch Available Patch Available Resolved Resolved
        7h 20m 1 Eli Collins 07/Sep/12 02:27
        Resolved Resolved Closed Closed
        34d 16h 17m 1 Arun C Murthy 11/Oct/12 18:45
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1189 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1189/)
        HADOOP-8431. Running distcp wo args throws IllegalArgumentException. Contributed by Sandy Ryza (Revision 1381840)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381840
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1189 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1189/ ) HADOOP-8431 . Running distcp wo args throws IllegalArgumentException. Contributed by Sandy Ryza (Revision 1381840) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381840 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1158 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1158/)
        HADOOP-8431. Running distcp wo args throws IllegalArgumentException. Contributed by Sandy Ryza (Revision 1381840)

        Result = FAILURE
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381840
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1158 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1158/ ) HADOOP-8431 . Running distcp wo args throws IllegalArgumentException. Contributed by Sandy Ryza (Revision 1381840) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381840 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java
        Hide
        Sho Shimauchi added a comment -

        Hi Eli and Sandy,

        I apologize for not having checked this issue carefully.
        Thanks for taking this and writing a patch!

        Show
        Sho Shimauchi added a comment - Hi Eli and Sandy, I apologize for not having checked this issue carefully. Thanks for taking this and writing a patch!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2725 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2725/)
        HADOOP-8431. Running distcp wo args throws IllegalArgumentException. Contributed by Sandy Ryza (Revision 1381840)

        Result = FAILURE
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381840
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2725 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2725/ ) HADOOP-8431 . Running distcp wo args throws IllegalArgumentException. Contributed by Sandy Ryza (Revision 1381840) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381840 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2701 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2701/)
        HADOOP-8431. Running distcp wo args throws IllegalArgumentException. Contributed by Sandy Ryza (Revision 1381840)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381840
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2701 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2701/ ) HADOOP-8431 . Running distcp wo args throws IllegalArgumentException. Contributed by Sandy Ryza (Revision 1381840) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381840 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2764 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2764/)
        HADOOP-8431. Running distcp wo args throws IllegalArgumentException. Contributed by Sandy Ryza (Revision 1381840)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381840
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2764 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2764/ ) HADOOP-8431 . Running distcp wo args throws IllegalArgumentException. Contributed by Sandy Ryza (Revision 1381840) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381840 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java
        Eli Collins made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Target Version/s 2.2.0-alpha [ 12322473 ]
        Fix Version/s 2.2.0-alpha [ 12322473 ]
        Resolution Fixed [ 1 ]
        Hide
        Eli Collins added a comment -

        I've committed this and merged to branch-2. Thanks Sandy!

        Show
        Eli Collins added a comment - I've committed this and merged to branch-2. Thanks Sandy!
        Hide
        Eli Collins added a comment -

        +1 looks great, verified the fix on a trunk tarball.

        Show
        Eli Collins added a comment - +1 looks great, verified the fix on a trunk tarball.
        Eli Collins made changes -
        Assignee Sandy Ryza [ sandyr ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12544091/diff1.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-distcp.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1415//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1415//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12544091/diff1.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-distcp. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1415//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1415//console This message is automatically generated.
        Sandy Ryza made changes -
        Attachment diff1.txt [ 12544091 ]
        Sandy Ryza made changes -
        Status Reopened [ 4 ] Patch Available [ 10002 ]
        Hide
        Eli Collins added a comment -

        It should just print the usage. Eg in the following I'd remove the ERROR log, the IllegalArgumentException backtrace and the "Invalid arguments" log.

        $ ./bin/hadoop distcp
        12/09/05 20:38:26 ERROR tools.DistCp: Invalid arguments: 
        java.lang.IllegalArgumentException: Target path not specified
        	at org.apache.hadoop.tools.OptionsParser.parse(OptionsParser.java:86)
        	at org.apache.hadoop.tools.DistCp.run(DistCp.java:102)
        	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        	at org.apache.hadoop.tools.DistCp.main(DistCp.java:368)
        Invalid arguments: Target path not specified
        usage: distcp OPTIONS [source_path...] <target_path>
                      OPTIONS
         -async                 Should distcp execution be blocking
         -atomic                Commit all changes or none
         -bandwidth <arg>       Specify bandwidth per map in MB
         -delete                Delete from target, files missing in source
         -f <arg>               List of files that need to be copied
         -filelimit <arg>       (Deprecated!) Limit number of files copied to <= n
         -i                     Ignore failures during copy
         -log <arg>             Folder on DFS where distcp execution logs are
                                saved
         -m <arg>               Max number of concurrent maps to use for copy
         -mapredSslConf <arg>   Configuration for ssl config file, to use with
                                hftps://
         -overwrite             Choose to overwrite target files unconditionally,
                                even if they exist.
         -p <arg>               preserve status (rbugp)(replication, block-size,
                                user, group, permission)
         -sizelimit <arg>       (Deprecated!) Limit number of files copied to <= n
                                bytes
         -skipcrccheck          Whether to skip CRC checks between source and
                                target paths.
         -strategy <arg>        Copy strategy to use. Default is dividing work
                                based on file sizes
         -tmp <arg>             Intermediate work path to be used for atomic
                                commit
         -update                Update target, copying only missingfiles or
                                directories
        
        Show
        Eli Collins added a comment - It should just print the usage. Eg in the following I'd remove the ERROR log, the IllegalArgumentException backtrace and the "Invalid arguments" log. $ ./bin/hadoop distcp 12/09/05 20:38:26 ERROR tools.DistCp: Invalid arguments: java.lang.IllegalArgumentException: Target path not specified at org.apache.hadoop.tools.OptionsParser.parse(OptionsParser.java:86) at org.apache.hadoop.tools.DistCp.run(DistCp.java:102) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.tools.DistCp.main(DistCp.java:368) Invalid arguments: Target path not specified usage: distcp OPTIONS [source_path...] <target_path> OPTIONS -async Should distcp execution be blocking -atomic Commit all changes or none -bandwidth <arg> Specify bandwidth per map in MB -delete Delete from target, files missing in source -f <arg> List of files that need to be copied -filelimit <arg> (Deprecated!) Limit number of files copied to <= n -i Ignore failures during copy -log <arg> Folder on DFS where distcp execution logs are saved -m <arg> Max number of concurrent maps to use for copy -mapredSslConf <arg> Configuration for ssl config file, to use with hftps:// -overwrite Choose to overwrite target files unconditionally, even if they exist. -p <arg> preserve status (rbugp)(replication, block-size, user, group, permission) -sizelimit <arg> (Deprecated!) Limit number of files copied to <= n bytes -skipcrccheck Whether to skip CRC checks between source and target paths. -strategy <arg> Copy strategy to use. Default is dividing work based on file sizes -tmp <arg> Intermediate work path to be used for atomic commit -update Update target, copying only missingfiles or directories
        Hide
        Sandy Ryza added a comment -

        I reproduced it as well. A usage() statement is printed in addition to the error - is the change to be made just removing the "Invalid arguments: Target path not specified" printed to System.err? Or the ERROR log statement as well?

        Show
        Sandy Ryza added a comment - I reproduced it as well. A usage() statement is printed in addition to the error - is the change to be made just removing the "Invalid arguments: Target path not specified" printed to System.err? Or the ERROR log statement as well?
        Eli Collins made changes -
        Target Version/s 2.1.0-alpha [ 12321441 ] 2.2.0-alpha [ 12322473 ]
        Eli Collins made changes -
        Resolution Cannot Reproduce [ 5 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Hide
        Eli Collins added a comment -

        I just tried this and confirmed it still fails with the latest build. In the future please try to reproduce the issue before you close it.

        hadoop-3.0.0-SNAPSHOT $ ./bin/hadoop distcp
        12/07/23 19:21:48 ERROR tools.DistCp: Invalid arguments:
        java.lang.IllegalArgumentException: Target path not specified
        at org.apache.hadoop.tools.OptionsParser.parse(OptionsParser.java:86)
        at org.apache.hadoop.tools.DistCp.run(DistCp.java:102)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.tools.DistCp.main(DistCp.java:368)

        Show
        Eli Collins added a comment - I just tried this and confirmed it still fails with the latest build. In the future please try to reproduce the issue before you close it. hadoop-3.0.0-SNAPSHOT $ ./bin/hadoop distcp 12/07/23 19:21:48 ERROR tools.DistCp: Invalid arguments: java.lang.IllegalArgumentException: Target path not specified at org.apache.hadoop.tools.OptionsParser.parse(OptionsParser.java:86) at org.apache.hadoop.tools.DistCp.run(DistCp.java:102) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.tools.DistCp.main(DistCp.java:368)
        Sho Shimauchi made changes -
        Field Original Value New Value
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Cannot Reproduce [ 5 ]
        Hide
        Sho Shimauchi added a comment -

        closing this JIRA.
        I've checked with Daisuke this issue is already fixed in MAPREDUCE-2765.
        Thanks for checking!

        Show
        Sho Shimauchi added a comment - closing this JIRA. I've checked with Daisuke this issue is already fixed in MAPREDUCE-2765 . Thanks for checking!
        Hide
        Daisuke Kobayashi added a comment -

        I'll close this issue.

        Show
        Daisuke Kobayashi added a comment - I'll close this issue.
        Hide
        Daisuke Kobayashi added a comment -

        I guess this has been fixed by MAPREDUCE-2765. If args are null, OptionsParser.usage() method would be called in DistCp.java.

        Show
        Daisuke Kobayashi added a comment - I guess this has been fixed by MAPREDUCE-2765 . If args are null, OptionsParser.usage() method would be called in DistCp.java.
        Hide
        Steve Loughran added a comment -

        What was the expected behaviour? A usage() statement?

        Show
        Steve Loughran added a comment - What was the expected behaviour? A usage() statement?
        Eli Collins created issue -

          People

          • Assignee:
            Sandy Ryza
            Reporter:
            Eli Collins
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development