Details

    • Type: Improvement Improvement
    • Status: Patch Available
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.6.0
    • Fix Version/s: None
    • Component/s: util
    • Labels:
    • Target Version/s:

      Description

      There should be a way to ignore specific paths (eg: those that have already been copied over under the current srcPath).

      1. HADOOP-1540.004.patch
        31 kB
        Rich Haase
      2. HADOOP-1540.003.patch
        19 kB
        Rich Haase

        Activity

        Senthil Subramanian created issue -
        Doug Cutting made changes -
        Field Original Value New Value
        Fix Version/s 0.14.0 [ 12312474 ]
        Hide
        Ravi Gummadi added a comment -

        Isn't -update option good enough for doing "ignore those that have already been copied over under the current srcPath" ?

        Show
        Ravi Gummadi added a comment - Isn't -update option good enough for doing "ignore those that have already been copied over under the current srcPath" ?
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Yes, we already have "-update". I think this is a stale issue. Closing this as "won't fix". Free feel to re-open it if this is a need.

        Show
        Tsz Wo Nicholas Sze added a comment - Yes, we already have "-update". I think this is a stale issue. Closing this as "won't fix". Free feel to re-open it if this is a need.
        Tsz Wo Nicholas Sze made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Won't Fix [ 2 ]
        Hide
        Laurent Edel added a comment -

        Event if this should have been implemented in https://issues.apache.org/jira/browse/MAPREDUCE-5014, it's apparently not.

        This would be very useful, for example not trying to copy Flume files that are not flushed yet (i.e. exclude .tmp files)

        Show
        Laurent Edel added a comment - Event if this should have been implemented in https://issues.apache.org/jira/browse/MAPREDUCE-5014 , it's apparently not. This would be very useful, for example not trying to copy Flume files that are not flushed yet (i.e. exclude .tmp files)
        Laurent Edel made changes -
        Resolution Won't Fix [ 2 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Hide
        Rich Haase added a comment -

        I have a patch for this JIRA that I've just started testing. https://github.com/richhaase/hadoop-patches/blob/master/HADOOP-1540.branch-2.6.0.001.patch

        The patch adds a -exclusions <arg> option to distcp. The argument is a file containing a list of Java Regex Patterns (one per line). Each file that is to be copied will be compared the list of exclusion patterns. IF an exclusion pattern is matched then the file will not be copied.

        Example CLI (running with a patched JAR on a Hortonworks HDP 2.2.4 cluster):

        $ export HADOOP_USER_CLASSPATH_FIRST=true; export HADOOP_CLASSPATH=/home/rhaase/hadoop-distcp-2.6.0-20150426160037.jar; mapred distcp -update -exclusions exclude.txt /user/hadoop/radio /user/rhaase/radio
        5/04/27 15:26:55 INFO tools.DistCp: Input Options: DistCpOptions

        {atomicCommit=false, syncFolder=true, deleteMissing=false, ignoreFailures=false, maxMaps=20, sslConfigurationFile='null', copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[/user/hadoop/radio], targetPath=/user/rhaase/radio, targetPathExists=false, preserveRawXattrs=false, exclusionsFile='exclude.txt'}

        ...
        15/04/27 15:42:27 INFO mapreduce.Job: map 100% reduce 0%
        15/04/27 15:42:27 INFO mapreduce.Job: Job job_1429896015201_0035 completed successfully
        15/04/27 15:42:27 INFO mapreduce.Job: Counters: 35
        File System Counters
        FILE: Number of bytes read=0
        FILE: Number of bytes written=2392499
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=358894362945
        HDFS: Number of bytes written=358893418844
        HDFS: Number of read operations=3214
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=633
        Job Counters
        Launched map tasks=21
        Other local map tasks=21
        Total time spent by all maps in occupied slots (ms)=4297461
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=4297461
        Total vcore-seconds taken by all map tasks=4297461
        Total megabyte-seconds taken by all map tasks=4400600064
        Map-Reduce Framework
        Map input records=4296
        Map output records=0
        Input split bytes=2457
        Spilled Records=0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=4573
        CPU time spent (ms)=2571060
        Physical memory (bytes) snapshot=10379874304
        Virtual memory (bytes) snapshot=56655720448
        Total committed heap usage (bytes)=43711463424
        File Input Format Counters
        Bytes Read=941644
        File Output Format Counters
        Bytes Written=0
        org.apache.hadoop.tools.mapred.CopyMapper$Counter
        BYTESCOPIED=358893418844
        BYTESEXCLUDED=1407553620118
        BYTESEXPECTED=358893418844
        COPY=322
        EXCLUDED=3974

        Show
        Rich Haase added a comment - I have a patch for this JIRA that I've just started testing. https://github.com/richhaase/hadoop-patches/blob/master/HADOOP-1540.branch-2.6.0.001.patch The patch adds a -exclusions <arg> option to distcp. The argument is a file containing a list of Java Regex Patterns (one per line). Each file that is to be copied will be compared the list of exclusion patterns. IF an exclusion pattern is matched then the file will not be copied. Example CLI (running with a patched JAR on a Hortonworks HDP 2.2.4 cluster): $ export HADOOP_USER_CLASSPATH_FIRST=true; export HADOOP_CLASSPATH=/home/rhaase/hadoop-distcp-2.6.0-20150426160037.jar; mapred distcp -update -exclusions exclude.txt /user/hadoop/radio /user/rhaase/radio 5/04/27 15:26:55 INFO tools.DistCp: Input Options: DistCpOptions {atomicCommit=false, syncFolder=true, deleteMissing=false, ignoreFailures=false, maxMaps=20, sslConfigurationFile='null', copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[/user/hadoop/radio], targetPath=/user/rhaase/radio, targetPathExists=false, preserveRawXattrs=false, exclusionsFile='exclude.txt'} ... 15/04/27 15:42:27 INFO mapreduce.Job: map 100% reduce 0% 15/04/27 15:42:27 INFO mapreduce.Job: Job job_1429896015201_0035 completed successfully 15/04/27 15:42:27 INFO mapreduce.Job: Counters: 35 File System Counters FILE: Number of bytes read=0 FILE: Number of bytes written=2392499 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=358894362945 HDFS: Number of bytes written=358893418844 HDFS: Number of read operations=3214 HDFS: Number of large read operations=0 HDFS: Number of write operations=633 Job Counters Launched map tasks=21 Other local map tasks=21 Total time spent by all maps in occupied slots (ms)=4297461 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=4297461 Total vcore-seconds taken by all map tasks=4297461 Total megabyte-seconds taken by all map tasks=4400600064 Map-Reduce Framework Map input records=4296 Map output records=0 Input split bytes=2457 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=4573 CPU time spent (ms)=2571060 Physical memory (bytes) snapshot=10379874304 Virtual memory (bytes) snapshot=56655720448 Total committed heap usage (bytes)=43711463424 File Input Format Counters Bytes Read=941644 File Output Format Counters Bytes Written=0 org.apache.hadoop.tools.mapred.CopyMapper$Counter BYTESCOPIED=358893418844 BYTESEXCLUDED=1407553620118 BYTESEXPECTED=358893418844 COPY=322 EXCLUDED=3974
        Hide
        Jing Zhao added a comment -

        Hi Rich Haase, do you want to upload your patch to this jira and run the Jenkins against it? We can help review the patch. Thanks!

        Show
        Jing Zhao added a comment - Hi Rich Haase , do you want to upload your patch to this jira and run the Jenkins against it? We can help review the patch. Thanks!
        Hide
        Rich Haase added a comment -

        Submitting patch for Jenkins test run. I think there is a bug in the way I am handling the argument to -exclusions. Files only can be read from the files system configured in hdfs-site.xml in the tests I've been running on an actual cluster.

        Show
        Rich Haase added a comment - Submitting patch for Jenkins test run. I think there is a bug in the way I am handling the argument to -exclusions. Files only can be read from the files system configured in hdfs-site.xml in the tests I've been running on an actual cluster.
        Rich Haase made changes -
        Status Reopened [ 4 ] Patch Available [ 10002 ]
        Affects Version/s 2.6.0 [ 12327179 ]
        Labels patch
        Assignee Rich Haase [ rhaase ]
        Fix Version/s 2.6.0 [ 12327179 ]
        Rich Haase made changes -
        Attachment HADOOP-1540.branch-2.6.0.001.patch [ 12728697 ]
        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 29s Pre-patch branch-2 compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        -1 patch 0m 0s The patch command could not apply the patch.



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12728697/HADOOP-1540.branch-2.6.0.001.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision branch-2 / a82addd
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6201/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 29s Pre-patch branch-2 compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 whitespace 0m 0s The patch has no lines that end in whitespace. -1 patch 0m 0s The patch command could not apply the patch. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12728697/HADOOP-1540.branch-2.6.0.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision branch-2 / a82addd Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6201/console This message was automatically generated.
        Hide
        Jing Zhao added a comment -

        Hi Rich Haase, thanks for uploading the patch! Do you mind rebasing the patch against the current trunk branch? 2.6 has already been released. And for new features and improvement we usually first commit them into trunk and then merge into branch-2 (which is currently aiming for 2.8).

        Show
        Jing Zhao added a comment - Hi Rich Haase , thanks for uploading the patch! Do you mind rebasing the patch against the current trunk branch? 2.6 has already been released. And for new features and improvement we usually first commit them into trunk and then merge into branch-2 (which is currently aiming for 2.8).
        Hide
        Rich Haase added a comment -

        Jing Zhao Just finished rebasing against trunk and testing the patch.

        Show
        Rich Haase added a comment - Jing Zhao Just finished rebasing against trunk and testing the patch.
        Hide
        Rich Haase added a comment -

        rebased patch against trunk

        Show
        Rich Haase added a comment - rebased patch against trunk
        Rich Haase made changes -
        Attachment HADOOP-1540.001.patch [ 12728904 ]
        Rich Haase made changes -
        Attachment HADOOP-1540.branch-2.6.0.001.patch [ 12728697 ]
        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 34s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        -1 javac 7m 30s The applied patch generated 1 additional warning messages.
        +1 javadoc 9m 35s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 5m 25s The applied patch generated 5 additional checkstyle issues.
        +1 install 1m 33s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        -1 findbugs 0m 43s The patch appears to introduce 2 new Findbugs (version 2.0.3) warnings.
        +1 tools/hadoop tests 6m 17s Tests passed in hadoop-distcp.
            46m 37s  



        Reason Tests
        FindBugs module:hadoop-distcp
          Dead store to localFS in org.apache.hadoop.tools.DistCp.addExclusionsFileToDistCache(Job, Path) At DistCp.java:org.apache.hadoop.tools.DistCp.addExclusionsFileToDistCache(Job, Path) At DistCp.java:[line 270]
          Found reliance on default encoding in org.apache.hadoop.tools.mapred.CopyMapper.initializeExclusionPatterns(Mapper$Context):in org.apache.hadoop.tools.mapred.CopyMapper.initializeExclusionPatterns(Mapper$Context): new java.io.InputStreamReader(InputStream) At CopyMapper.java:[line 163]



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12728904/HADOOP-1540.001.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 5639bf0
        javac https://builds.apache.org/job/PreCommit-HADOOP-Build/6207/artifact/patchprocess/diffJavacWarnings.txt
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6207/artifact/patchprocess/checkstyle-result-diff.txt
        Findbugs warnings https://builds.apache.org/job/PreCommit-HADOOP-Build/6207/artifact/patchprocess/newPatchFindbugsWarningshadoop-distcp.html
        hadoop-distcp test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6207/artifact/patchprocess/testrun_hadoop-distcp.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6207/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6207/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 34s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 whitespace 0m 0s The patch has no lines that end in whitespace. -1 javac 7m 30s The applied patch generated 1 additional warning messages. +1 javadoc 9m 35s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 5m 25s The applied patch generated 5 additional checkstyle issues. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. -1 findbugs 0m 43s The patch appears to introduce 2 new Findbugs (version 2.0.3) warnings. +1 tools/hadoop tests 6m 17s Tests passed in hadoop-distcp.     46m 37s   Reason Tests FindBugs module:hadoop-distcp   Dead store to localFS in org.apache.hadoop.tools.DistCp.addExclusionsFileToDistCache(Job, Path) At DistCp.java:org.apache.hadoop.tools.DistCp.addExclusionsFileToDistCache(Job, Path) At DistCp.java: [line 270]   Found reliance on default encoding in org.apache.hadoop.tools.mapred.CopyMapper.initializeExclusionPatterns(Mapper$Context):in org.apache.hadoop.tools.mapred.CopyMapper.initializeExclusionPatterns(Mapper$Context): new java.io.InputStreamReader(InputStream) At CopyMapper.java: [line 163] Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12728904/HADOOP-1540.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 5639bf0 javac https://builds.apache.org/job/PreCommit-HADOOP-Build/6207/artifact/patchprocess/diffJavacWarnings.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6207/artifact/patchprocess/checkstyle-result-diff.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HADOOP-Build/6207/artifact/patchprocess/newPatchFindbugsWarningshadoop-distcp.html hadoop-distcp test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6207/artifact/patchprocess/testrun_hadoop-distcp.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6207/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6207/console This message was automatically generated.
        Hide
        Rich Haase added a comment -

        Working on fixing the items Jenkins is complaining about.

        Show
        Rich Haase added a comment - Working on fixing the items Jenkins is complaining about.
        Hide
        Rich Haase added a comment -

        This revision of the patch should fix findbugs/javac warnings.

        Show
        Rich Haase added a comment - This revision of the patch should fix findbugs/javac warnings.
        Rich Haase made changes -
        Attachment HADOOP-1540.002.patch [ 12728958 ]
        Rich Haase made changes -
        Attachment HADOOP-1540.001.patch [ 12728904 ]
        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 54s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 javac 7m 37s There were no new javac warning messages.
        +1 javadoc 9m 50s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 4m 1s The applied patch generated 5 additional checkstyle issues.
        +1 install 1m 35s mvn install still works.
        +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
        +1 findbugs 0m 38s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 tools/hadoop tests 6m 20s Tests passed in hadoop-distcp.
            45m 59s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12728958/HADOOP-1540.002.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 5190923
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6208/artifact/patchprocess/checkstyle-result-diff.txt
        hadoop-distcp test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6208/artifact/patchprocess/testrun_hadoop-distcp.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6208/testReport/
        Java 1.7.0_55
        uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6208/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 54s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 javac 7m 37s There were no new javac warning messages. +1 javadoc 9m 50s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 4m 1s The applied patch generated 5 additional checkstyle issues. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. +1 findbugs 0m 38s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 tools/hadoop tests 6m 20s Tests passed in hadoop-distcp.     45m 59s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12728958/HADOOP-1540.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 5190923 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6208/artifact/patchprocess/checkstyle-result-diff.txt hadoop-distcp test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6208/artifact/patchprocess/testrun_hadoop-distcp.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6208/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6208/console This message was automatically generated.
        Hide
        Rich Haase added a comment -

        Fixed checkstyle errors

        Show
        Rich Haase added a comment - Fixed checkstyle errors
        Rich Haase made changes -
        Attachment HADOOP-1540.003.patch [ 12729842 ]
        Rich Haase made changes -
        Attachment HADOOP-1540.002.patch [ 12728958 ]
        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 15m 2s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 javac 7m 45s There were no new javac warning messages.
        +1 javadoc 9m 55s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 0m 28s The applied patch generated 1 new checkstyle issues (total was 162, now 161).
        +1 whitespace 0m 1s The patch has no lines that end in whitespace.
        +1 install 1m 34s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 0m 41s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 tools/hadoop tests 6m 12s Tests passed in hadoop-distcp.
            42m 42s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12729842/HADOOP-1540.003.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / d3d019c
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6243/artifact/patchprocess/diffcheckstylehadoop-distcp.txt
        hadoop-distcp test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6243/artifact/patchprocess/testrun_hadoop-distcp.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6243/testReport/
        Java 1.7.0_55
        uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6243/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 2s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 45s There were no new javac warning messages. +1 javadoc 9m 55s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 28s The applied patch generated 1 new checkstyle issues (total was 162, now 161). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 0m 41s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 tools/hadoop tests 6m 12s Tests passed in hadoop-distcp.     42m 42s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12729842/HADOOP-1540.003.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / d3d019c checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6243/artifact/patchprocess/diffcheckstylehadoop-distcp.txt hadoop-distcp test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6243/artifact/patchprocess/testrun_hadoop-distcp.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6243/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6243/console This message was automatically generated.
        Hide
        Rich Haase added a comment -

        I'm not sure what to do about this final checkstyle warning:

        ./hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java:77:3: Method length is 196 lines (max allowed is 150).

        The method in question is parse(), which was about 190 lines before I made my changes and is now 196 lines. I can break up the parse method, but that seems like it would be more appropriate if this were a refactoring change, rather than a feature addition. Can someone offer some suggestions for how I should handle this?

        Show
        Rich Haase added a comment - I'm not sure what to do about this final checkstyle warning: ./hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java:77:3: Method length is 196 lines (max allowed is 150). The method in question is parse(), which was about 190 lines before I made my changes and is now 196 lines. I can break up the parse method, but that seems like it would be more appropriate if this were a refactoring change, rather than a feature addition. Can someone offer some suggestions for how I should handle this?
        Hide
        Rich Haase added a comment -

        Take 4. I think I have all of the checkstyle issues fixed. I want to object super strongly to an 80 char limit for line length. It's been a long time since 80 chars was a reasonable line length.

        Show
        Rich Haase added a comment - Take 4. I think I have all of the checkstyle issues fixed. I want to object super strongly to an 80 char limit for line length. It's been a long time since 80 chars was a reasonable line length.
        Rich Haase made changes -
        Attachment HADOOP-1540.004.patch [ 12730587 ]
        Hide
        Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 16m 56s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 javac 8m 44s There were no new javac warning messages.
        +1 javadoc 9m 44s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 32s There were no new checkstyle issues.
        +1 whitespace 0m 3s The patch has no lines that end in whitespace.
        +1 install 1m 34s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 0m 38s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 tools/hadoop tests 6m 15s Tests passed in hadoop-distcp.
            45m 32s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12730587/HADOOP-1540.004.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / ffce9a3
        hadoop-distcp test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6487/artifact/patchprocess/testrun_hadoop-distcp.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6487/testReport/
        Java 1.7.0_55
        uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6487/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 56s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 8m 44s There were no new javac warning messages. +1 javadoc 9m 44s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 32s There were no new checkstyle issues. +1 whitespace 0m 3s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 0m 38s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 tools/hadoop tests 6m 15s Tests passed in hadoop-distcp.     45m 32s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12730587/HADOOP-1540.004.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / ffce9a3 hadoop-distcp test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6487/artifact/patchprocess/testrun_hadoop-distcp.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6487/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6487/console This message was automatically generated.
        Rich Haase made changes -
        Fix Version/s 2.6.0 [ 12327179 ]
        Target Version/s 3.0.0 [ 12320357 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        719d 22h 12m 1 Tsz Wo Nicholas Sze 16/Jun/09 21:43
        Resolved Resolved Reopened Reopened
        2033d 3h 58m 1 Laurent Edel 10/Jan/15 00:41
        Reopened Reopened Patch Available Patch Available
        108d 4h 8m 1 Rich Haase 28/Apr/15 05:50

          People

          • Assignee:
            Rich Haase
            Reporter:
            Senthil Subramanian
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:

              Development