Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13034

Log message about input options in distcp lacks some items

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: tools/distcp
    • Labels:
      None

      Description

      The log message in running distcp does not show some options, i.e. append, useDiff and snapshot names.

      16/04/18 21:57:36 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=true, deleteMissing=false, ignoreFailures=false, overwrite=false, skipCRC=false, blocking=true, numListstatusThreads=0, maxMaps=20, mapBandwidth=100.0, copyStrategy='uniformsize', preserveStatus=[], preserveRawXattrs=false, atomicWorkPath=null, logPath=null, sourceFileListing=null, sourcePaths=[/user/hadoop/source], targetPath=/user/hadoop/target, targetPathExists=true, filtersFile='null'}
      

      I think that this message is useful for debugging and so it is better to add the lacked options.

      1. HADOOP-13034.1.patch
        2 kB
        Takashi Ohnishi

        Activity

        Hide
        yzhangal Yongjun Zhang added a comment -

        Thanks you guys for the earlier work, I cherry-picked to branch-2.

        Show
        yzhangal Yongjun Zhang added a comment - Thanks you guys for the earlier work, I cherry-picked to branch-2.
        Hide
        bwtakacy Takashi Ohnishi added a comment -

        Thank you Allen Wittenauer for committing!
        Thank you Daniel Templeton and Steve Loughran for reveiwing!!

        Show
        bwtakacy Takashi Ohnishi added a comment - Thank you Allen Wittenauer for committing! Thank you Daniel Templeton and Steve Loughran for reveiwing!!
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 37s trunk passed
        +1 compile 0m 17s trunk passed
        +1 checkstyle 0m 14s trunk passed
        +1 mvnsite 0m 21s trunk passed
        +1 mvneclipse 0m 16s trunk passed
        +1 findbugs 0m 30s trunk passed
        +1 javadoc 0m 12s trunk passed
        +1 mvninstall 0m 16s the patch passed
        +1 compile 0m 15s the patch passed
        +1 javac 0m 15s the patch passed
        +1 checkstyle 0m 11s the patch passed
        +1 mvnsite 0m 18s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 29s the patch passed
        +1 javadoc 0m 10s the patch passed
        +1 unit 9m 32s hadoop-distcp in the patch passed.
        +1 asflicense 0m 14s The patch does not generate ASF License warnings.
        21m 38s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:85209cc
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799247/HADOOP-13034.1.patch
        JIRA Issue HADOOP-13034
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 0710bfa05f9b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 23c3ff8
        Default Java 1.8.0_91
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9892/testReport/
        modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9892/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 37s trunk passed +1 compile 0m 17s trunk passed +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 21s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 0m 30s trunk passed +1 javadoc 0m 12s trunk passed +1 mvninstall 0m 16s the patch passed +1 compile 0m 15s the patch passed +1 javac 0m 15s the patch passed +1 checkstyle 0m 11s the patch passed +1 mvnsite 0m 18s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 29s the patch passed +1 javadoc 0m 10s the patch passed +1 unit 9m 32s hadoop-distcp in the patch passed. +1 asflicense 0m 14s The patch does not generate ASF License warnings. 21m 38s Subsystem Report/Notes Docker Image:yetus/hadoop:85209cc JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799247/HADOOP-13034.1.patch JIRA Issue HADOOP-13034 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0710bfa05f9b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 23c3ff8 Default Java 1.8.0_91 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9892/testReport/ modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9892/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #10027 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10027/)
        HADOOP-13034. Log message about input options in distcp lacks some items (aw: rev 422c73a8657d8699920f7db13d4be200e16c4272)

        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #10027 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10027/ ) HADOOP-13034 . Log message about input options in distcp lacks some items (aw: rev 422c73a8657d8699920f7db13d4be200e16c4272) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
        Hide
        aw Allen Wittenauer added a comment -

        +1 committed to trunk.

        Thanks!

        Show
        aw Allen Wittenauer added a comment - +1 committed to trunk. Thanks!
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 11s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 56s trunk passed
        +1 compile 0m 17s trunk passed
        +1 checkstyle 0m 14s trunk passed
        +1 mvnsite 0m 21s trunk passed
        +1 mvneclipse 0m 46s trunk passed
        +1 findbugs 0m 24s trunk passed
        +1 javadoc 0m 13s trunk passed
        +1 mvninstall 0m 15s the patch passed
        +1 compile 0m 14s the patch passed
        +1 javac 0m 14s the patch passed
        +1 checkstyle 0m 11s the patch passed
        +1 mvnsite 0m 18s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 28s the patch passed
        +1 javadoc 0m 10s the patch passed
        +1 unit 9m 15s hadoop-distcp in the patch passed.
        +1 asflicense 0m 15s The patch does not generate ASF License warnings.
        21m 52s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:85209cc
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799247/HADOOP-13034.1.patch
        JIRA Issue HADOOP-13034
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 0c5a2574f2fc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 23c3ff8
        Default Java 1.8.0_91
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9886/testReport/
        modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9886/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 56s trunk passed +1 compile 0m 17s trunk passed +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 21s trunk passed +1 mvneclipse 0m 46s trunk passed +1 findbugs 0m 24s trunk passed +1 javadoc 0m 13s trunk passed +1 mvninstall 0m 15s the patch passed +1 compile 0m 14s the patch passed +1 javac 0m 14s the patch passed +1 checkstyle 0m 11s the patch passed +1 mvnsite 0m 18s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 28s the patch passed +1 javadoc 0m 10s the patch passed +1 unit 9m 15s hadoop-distcp in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 21m 52s Subsystem Report/Notes Docker Image:yetus/hadoop:85209cc JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799247/HADOOP-13034.1.patch JIRA Issue HADOOP-13034 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0c5a2574f2fc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 23c3ff8 Default Java 1.8.0_91 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9886/testReport/ modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9886/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        stevel@apache.org Steve Loughran added a comment -

        +1

        Show
        stevel@apache.org Steve Loughran added a comment - +1
        Hide
        templedf Daniel Templeton added a comment -

        LGTM. Looks like you got all the ones that were missing.

        Show
        templedf Daniel Templeton added a comment - LGTM. Looks like you got all the ones that were missing.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 9m 41s trunk passed
        +1 compile 0m 28s trunk passed with JDK v1.8.0_77
        +1 compile 0m 25s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 22s trunk passed
        +1 mvnsite 0m 32s trunk passed
        +1 mvneclipse 0m 21s trunk passed
        +1 findbugs 0m 41s trunk passed
        +1 javadoc 0m 24s trunk passed with JDK v1.8.0_77
        +1 javadoc 0m 22s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 24s the patch passed
        +1 compile 0m 23s the patch passed with JDK v1.8.0_77
        +1 javac 0m 23s the patch passed
        +1 compile 0m 22s the patch passed with JDK v1.7.0_95
        +1 javac 0m 22s the patch passed
        +1 checkstyle 0m 16s the patch passed
        +1 mvnsite 0m 27s the patch passed
        +1 mvneclipse 0m 15s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 52s the patch passed
        +1 javadoc 0m 20s the patch passed with JDK v1.8.0_77
        +1 javadoc 0m 18s the patch passed with JDK v1.7.0_95
        +1 unit 10m 41s hadoop-distcp in the patch passed with JDK v1.8.0_77.
        +1 unit 9m 54s hadoop-distcp in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 29s Patch does not generate ASF License warnings.
        39m 39s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799247/HADOOP-13034.1.patch
        JIRA Issue HADOOP-13034
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 45af45862b20 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d8b729e
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9110/testReport/
        modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9110/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 9m 41s trunk passed +1 compile 0m 28s trunk passed with JDK v1.8.0_77 +1 compile 0m 25s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 22s trunk passed +1 mvnsite 0m 32s trunk passed +1 mvneclipse 0m 21s trunk passed +1 findbugs 0m 41s trunk passed +1 javadoc 0m 24s trunk passed with JDK v1.8.0_77 +1 javadoc 0m 22s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 24s the patch passed +1 compile 0m 23s the patch passed with JDK v1.8.0_77 +1 javac 0m 23s the patch passed +1 compile 0m 22s the patch passed with JDK v1.7.0_95 +1 javac 0m 22s the patch passed +1 checkstyle 0m 16s the patch passed +1 mvnsite 0m 27s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 52s the patch passed +1 javadoc 0m 20s the patch passed with JDK v1.8.0_77 +1 javadoc 0m 18s the patch passed with JDK v1.7.0_95 +1 unit 10m 41s hadoop-distcp in the patch passed with JDK v1.8.0_77. +1 unit 9m 54s hadoop-distcp in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 29s Patch does not generate ASF License warnings. 39m 39s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799247/HADOOP-13034.1.patch JIRA Issue HADOOP-13034 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 45af45862b20 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d8b729e Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9110/testReport/ modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9110/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        bwtakacy Takashi Ohnishi added a comment -

        Attached patch.

        Show
        bwtakacy Takashi Ohnishi added a comment - Attached patch.

          People

          • Assignee:
            bwtakacy Takashi Ohnishi
            Reporter:
            bwtakacy Takashi Ohnishi
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development