Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.8.0
    • Component/s: util
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      There should be a way to ignore specific paths (eg: those that have already been copied over under the current srcPath).

        Activity

        Hide
        Ravi Gummadi added a comment -

        Isn't -update option good enough for doing "ignore those that have already been copied over under the current srcPath" ?

        Show
        Ravi Gummadi added a comment - Isn't -update option good enough for doing "ignore those that have already been copied over under the current srcPath" ?
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Yes, we already have "-update". I think this is a stale issue. Closing this as "won't fix". Free feel to re-open it if this is a need.

        Show
        Tsz Wo Nicholas Sze added a comment - Yes, we already have "-update". I think this is a stale issue. Closing this as "won't fix". Free feel to re-open it if this is a need.
        Hide
        Laurent Edel added a comment -

        Event if this should have been implemented in https://issues.apache.org/jira/browse/MAPREDUCE-5014, it's apparently not.

        This would be very useful, for example not trying to copy Flume files that are not flushed yet (i.e. exclude .tmp files)

        Show
        Laurent Edel added a comment - Event if this should have been implemented in https://issues.apache.org/jira/browse/MAPREDUCE-5014 , it's apparently not. This would be very useful, for example not trying to copy Flume files that are not flushed yet (i.e. exclude .tmp files)
        Hide
        Rich Haase added a comment -

        I have a patch for this JIRA that I've just started testing. https://github.com/richhaase/hadoop-patches/blob/master/HADOOP-1540.branch-2.6.0.001.patch

        The patch adds a -exclusions <arg> option to distcp. The argument is a file containing a list of Java Regex Patterns (one per line). Each file that is to be copied will be compared the list of exclusion patterns. IF an exclusion pattern is matched then the file will not be copied.

        Example CLI (running with a patched JAR on a Hortonworks HDP 2.2.4 cluster):

        $ export HADOOP_USER_CLASSPATH_FIRST=true; export HADOOP_CLASSPATH=/home/rhaase/hadoop-distcp-2.6.0-20150426160037.jar; mapred distcp -update -exclusions exclude.txt /user/hadoop/radio /user/rhaase/radio
        5/04/27 15:26:55 INFO tools.DistCp: Input Options: DistCpOptions

        {atomicCommit=false, syncFolder=true, deleteMissing=false, ignoreFailures=false, maxMaps=20, sslConfigurationFile='null', copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[/user/hadoop/radio], targetPath=/user/rhaase/radio, targetPathExists=false, preserveRawXattrs=false, exclusionsFile='exclude.txt'}

        ...
        15/04/27 15:42:27 INFO mapreduce.Job: map 100% reduce 0%
        15/04/27 15:42:27 INFO mapreduce.Job: Job job_1429896015201_0035 completed successfully
        15/04/27 15:42:27 INFO mapreduce.Job: Counters: 35
        File System Counters
        FILE: Number of bytes read=0
        FILE: Number of bytes written=2392499
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=358894362945
        HDFS: Number of bytes written=358893418844
        HDFS: Number of read operations=3214
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=633
        Job Counters
        Launched map tasks=21
        Other local map tasks=21
        Total time spent by all maps in occupied slots (ms)=4297461
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=4297461
        Total vcore-seconds taken by all map tasks=4297461
        Total megabyte-seconds taken by all map tasks=4400600064
        Map-Reduce Framework
        Map input records=4296
        Map output records=0
        Input split bytes=2457
        Spilled Records=0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=4573
        CPU time spent (ms)=2571060
        Physical memory (bytes) snapshot=10379874304
        Virtual memory (bytes) snapshot=56655720448
        Total committed heap usage (bytes)=43711463424
        File Input Format Counters
        Bytes Read=941644
        File Output Format Counters
        Bytes Written=0
        org.apache.hadoop.tools.mapred.CopyMapper$Counter
        BYTESCOPIED=358893418844
        BYTESEXCLUDED=1407553620118
        BYTESEXPECTED=358893418844
        COPY=322
        EXCLUDED=3974

        Show
        Rich Haase added a comment - I have a patch for this JIRA that I've just started testing. https://github.com/richhaase/hadoop-patches/blob/master/HADOOP-1540.branch-2.6.0.001.patch The patch adds a -exclusions <arg> option to distcp. The argument is a file containing a list of Java Regex Patterns (one per line). Each file that is to be copied will be compared the list of exclusion patterns. IF an exclusion pattern is matched then the file will not be copied. Example CLI (running with a patched JAR on a Hortonworks HDP 2.2.4 cluster): $ export HADOOP_USER_CLASSPATH_FIRST=true; export HADOOP_CLASSPATH=/home/rhaase/hadoop-distcp-2.6.0-20150426160037.jar; mapred distcp -update -exclusions exclude.txt /user/hadoop/radio /user/rhaase/radio 5/04/27 15:26:55 INFO tools.DistCp: Input Options: DistCpOptions {atomicCommit=false, syncFolder=true, deleteMissing=false, ignoreFailures=false, maxMaps=20, sslConfigurationFile='null', copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[/user/hadoop/radio], targetPath=/user/rhaase/radio, targetPathExists=false, preserveRawXattrs=false, exclusionsFile='exclude.txt'} ... 15/04/27 15:42:27 INFO mapreduce.Job: map 100% reduce 0% 15/04/27 15:42:27 INFO mapreduce.Job: Job job_1429896015201_0035 completed successfully 15/04/27 15:42:27 INFO mapreduce.Job: Counters: 35 File System Counters FILE: Number of bytes read=0 FILE: Number of bytes written=2392499 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=358894362945 HDFS: Number of bytes written=358893418844 HDFS: Number of read operations=3214 HDFS: Number of large read operations=0 HDFS: Number of write operations=633 Job Counters Launched map tasks=21 Other local map tasks=21 Total time spent by all maps in occupied slots (ms)=4297461 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=4297461 Total vcore-seconds taken by all map tasks=4297461 Total megabyte-seconds taken by all map tasks=4400600064 Map-Reduce Framework Map input records=4296 Map output records=0 Input split bytes=2457 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=4573 CPU time spent (ms)=2571060 Physical memory (bytes) snapshot=10379874304 Virtual memory (bytes) snapshot=56655720448 Total committed heap usage (bytes)=43711463424 File Input Format Counters Bytes Read=941644 File Output Format Counters Bytes Written=0 org.apache.hadoop.tools.mapred.CopyMapper$Counter BYTESCOPIED=358893418844 BYTESEXCLUDED=1407553620118 BYTESEXPECTED=358893418844 COPY=322 EXCLUDED=3974
        Hide
        Jing Zhao added a comment -

        Hi Rich Haase, do you want to upload your patch to this jira and run the Jenkins against it? We can help review the patch. Thanks!

        Show
        Jing Zhao added a comment - Hi Rich Haase , do you want to upload your patch to this jira and run the Jenkins against it? We can help review the patch. Thanks!
        Hide
        Rich Haase added a comment -

        Submitting patch for Jenkins test run. I think there is a bug in the way I am handling the argument to -exclusions. Files only can be read from the files system configured in hdfs-site.xml in the tests I've been running on an actual cluster.

        Show
        Rich Haase added a comment - Submitting patch for Jenkins test run. I think there is a bug in the way I am handling the argument to -exclusions. Files only can be read from the files system configured in hdfs-site.xml in the tests I've been running on an actual cluster.
        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 29s Pre-patch branch-2 compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        -1 patch 0m 0s The patch command could not apply the patch.



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12728697/HADOOP-1540.branch-2.6.0.001.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision branch-2 / a82addd
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6201/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 29s Pre-patch branch-2 compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 whitespace 0m 0s The patch has no lines that end in whitespace. -1 patch 0m 0s The patch command could not apply the patch. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12728697/HADOOP-1540.branch-2.6.0.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision branch-2 / a82addd Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6201/console This message was automatically generated.
        Hide
        Jing Zhao added a comment -

        Hi Rich Haase, thanks for uploading the patch! Do you mind rebasing the patch against the current trunk branch? 2.6 has already been released. And for new features and improvement we usually first commit them into trunk and then merge into branch-2 (which is currently aiming for 2.8).

        Show
        Jing Zhao added a comment - Hi Rich Haase , thanks for uploading the patch! Do you mind rebasing the patch against the current trunk branch? 2.6 has already been released. And for new features and improvement we usually first commit them into trunk and then merge into branch-2 (which is currently aiming for 2.8).
        Hide
        Rich Haase added a comment -

        Jing Zhao Just finished rebasing against trunk and testing the patch.

        Show
        Rich Haase added a comment - Jing Zhao Just finished rebasing against trunk and testing the patch.
        Hide
        Rich Haase added a comment -

        rebased patch against trunk

        Show
        Rich Haase added a comment - rebased patch against trunk
        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 34s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        -1 javac 7m 30s The applied patch generated 1 additional warning messages.
        +1 javadoc 9m 35s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 5m 25s The applied patch generated 5 additional checkstyle issues.
        +1 install 1m 33s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        -1 findbugs 0m 43s The patch appears to introduce 2 new Findbugs (version 2.0.3) warnings.
        +1 tools/hadoop tests 6m 17s Tests passed in hadoop-distcp.
            46m 37s  



        Reason Tests
        FindBugs module:hadoop-distcp
          Dead store to localFS in org.apache.hadoop.tools.DistCp.addExclusionsFileToDistCache(Job, Path) At DistCp.java:org.apache.hadoop.tools.DistCp.addExclusionsFileToDistCache(Job, Path) At DistCp.java:[line 270]
          Found reliance on default encoding in org.apache.hadoop.tools.mapred.CopyMapper.initializeExclusionPatterns(Mapper$Context):in org.apache.hadoop.tools.mapred.CopyMapper.initializeExclusionPatterns(Mapper$Context): new java.io.InputStreamReader(InputStream) At CopyMapper.java:[line 163]



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12728904/HADOOP-1540.001.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 5639bf0
        javac https://builds.apache.org/job/PreCommit-HADOOP-Build/6207/artifact/patchprocess/diffJavacWarnings.txt
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6207/artifact/patchprocess/checkstyle-result-diff.txt
        Findbugs warnings https://builds.apache.org/job/PreCommit-HADOOP-Build/6207/artifact/patchprocess/newPatchFindbugsWarningshadoop-distcp.html
        hadoop-distcp test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6207/artifact/patchprocess/testrun_hadoop-distcp.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6207/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6207/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 34s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 whitespace 0m 0s The patch has no lines that end in whitespace. -1 javac 7m 30s The applied patch generated 1 additional warning messages. +1 javadoc 9m 35s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 5m 25s The applied patch generated 5 additional checkstyle issues. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. -1 findbugs 0m 43s The patch appears to introduce 2 new Findbugs (version 2.0.3) warnings. +1 tools/hadoop tests 6m 17s Tests passed in hadoop-distcp.     46m 37s   Reason Tests FindBugs module:hadoop-distcp   Dead store to localFS in org.apache.hadoop.tools.DistCp.addExclusionsFileToDistCache(Job, Path) At DistCp.java:org.apache.hadoop.tools.DistCp.addExclusionsFileToDistCache(Job, Path) At DistCp.java: [line 270]   Found reliance on default encoding in org.apache.hadoop.tools.mapred.CopyMapper.initializeExclusionPatterns(Mapper$Context):in org.apache.hadoop.tools.mapred.CopyMapper.initializeExclusionPatterns(Mapper$Context): new java.io.InputStreamReader(InputStream) At CopyMapper.java: [line 163] Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12728904/HADOOP-1540.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 5639bf0 javac https://builds.apache.org/job/PreCommit-HADOOP-Build/6207/artifact/patchprocess/diffJavacWarnings.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6207/artifact/patchprocess/checkstyle-result-diff.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HADOOP-Build/6207/artifact/patchprocess/newPatchFindbugsWarningshadoop-distcp.html hadoop-distcp test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6207/artifact/patchprocess/testrun_hadoop-distcp.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6207/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6207/console This message was automatically generated.
        Hide
        Rich Haase added a comment -

        Working on fixing the items Jenkins is complaining about.

        Show
        Rich Haase added a comment - Working on fixing the items Jenkins is complaining about.
        Hide
        Rich Haase added a comment -

        This revision of the patch should fix findbugs/javac warnings.

        Show
        Rich Haase added a comment - This revision of the patch should fix findbugs/javac warnings.
        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 54s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 javac 7m 37s There were no new javac warning messages.
        +1 javadoc 9m 50s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 4m 1s The applied patch generated 5 additional checkstyle issues.
        +1 install 1m 35s mvn install still works.
        +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
        +1 findbugs 0m 38s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 tools/hadoop tests 6m 20s Tests passed in hadoop-distcp.
            45m 59s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12728958/HADOOP-1540.002.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 5190923
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6208/artifact/patchprocess/checkstyle-result-diff.txt
        hadoop-distcp test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6208/artifact/patchprocess/testrun_hadoop-distcp.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6208/testReport/
        Java 1.7.0_55
        uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6208/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 54s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 javac 7m 37s There were no new javac warning messages. +1 javadoc 9m 50s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 4m 1s The applied patch generated 5 additional checkstyle issues. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. +1 findbugs 0m 38s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 tools/hadoop tests 6m 20s Tests passed in hadoop-distcp.     45m 59s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12728958/HADOOP-1540.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 5190923 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6208/artifact/patchprocess/checkstyle-result-diff.txt hadoop-distcp test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6208/artifact/patchprocess/testrun_hadoop-distcp.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6208/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6208/console This message was automatically generated.
        Hide
        Rich Haase added a comment -

        Fixed checkstyle errors

        Show
        Rich Haase added a comment - Fixed checkstyle errors
        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 15m 2s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 javac 7m 45s There were no new javac warning messages.
        +1 javadoc 9m 55s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 0m 28s The applied patch generated 1 new checkstyle issues (total was 162, now 161).
        +1 whitespace 0m 1s The patch has no lines that end in whitespace.
        +1 install 1m 34s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 0m 41s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 tools/hadoop tests 6m 12s Tests passed in hadoop-distcp.
            42m 42s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12729842/HADOOP-1540.003.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / d3d019c
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6243/artifact/patchprocess/diffcheckstylehadoop-distcp.txt
        hadoop-distcp test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6243/artifact/patchprocess/testrun_hadoop-distcp.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6243/testReport/
        Java 1.7.0_55
        uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6243/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 2s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 45s There were no new javac warning messages. +1 javadoc 9m 55s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 28s The applied patch generated 1 new checkstyle issues (total was 162, now 161). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 0m 41s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 tools/hadoop tests 6m 12s Tests passed in hadoop-distcp.     42m 42s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12729842/HADOOP-1540.003.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / d3d019c checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6243/artifact/patchprocess/diffcheckstylehadoop-distcp.txt hadoop-distcp test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6243/artifact/patchprocess/testrun_hadoop-distcp.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6243/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6243/console This message was automatically generated.
        Hide
        Rich Haase added a comment -

        I'm not sure what to do about this final checkstyle warning:

        ./hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java:77:3: Method length is 196 lines (max allowed is 150).

        The method in question is parse(), which was about 190 lines before I made my changes and is now 196 lines. I can break up the parse method, but that seems like it would be more appropriate if this were a refactoring change, rather than a feature addition. Can someone offer some suggestions for how I should handle this?

        Show
        Rich Haase added a comment - I'm not sure what to do about this final checkstyle warning: ./hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java:77:3: Method length is 196 lines (max allowed is 150). The method in question is parse(), which was about 190 lines before I made my changes and is now 196 lines. I can break up the parse method, but that seems like it would be more appropriate if this were a refactoring change, rather than a feature addition. Can someone offer some suggestions for how I should handle this?
        Hide
        Rich Haase added a comment -

        Take 4. I think I have all of the checkstyle issues fixed. I want to object super strongly to an 80 char limit for line length. It's been a long time since 80 chars was a reasonable line length.

        Show
        Rich Haase added a comment - Take 4. I think I have all of the checkstyle issues fixed. I want to object super strongly to an 80 char limit for line length. It's been a long time since 80 chars was a reasonable line length.
        Hide
        Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 16m 56s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 javac 8m 44s There were no new javac warning messages.
        +1 javadoc 9m 44s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 32s There were no new checkstyle issues.
        +1 whitespace 0m 3s The patch has no lines that end in whitespace.
        +1 install 1m 34s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 0m 38s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 tools/hadoop tests 6m 15s Tests passed in hadoop-distcp.
            45m 32s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12730587/HADOOP-1540.004.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / ffce9a3
        hadoop-distcp test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6487/artifact/patchprocess/testrun_hadoop-distcp.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6487/testReport/
        Java 1.7.0_55
        uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6487/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 56s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 8m 44s There were no new javac warning messages. +1 javadoc 9m 44s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 32s There were no new checkstyle issues. +1 whitespace 0m 3s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 0m 38s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 tools/hadoop tests 6m 15s Tests passed in hadoop-distcp.     45m 32s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12730587/HADOOP-1540.004.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / ffce9a3 hadoop-distcp test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6487/artifact/patchprocess/testrun_hadoop-distcp.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6487/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6487/console This message was automatically generated.
        Hide
        Jing Zhao added a comment -

        Thanks again for working on this, Rich Haase! The patch looks good to me overall. Some comments and thoughts:

        1. In the current patch we expect the user to only define regular expressions for exclusions. However, since the patch allows the user to define all the patterns into a file, it is possible that the user puts a long list of file names into the exclusion file (by writing some script e.g.), which can cause issue since the Mapper will compile every line into a regex pattern. Thus I guess what we can do here is to limit the total number of regex. Another option can be to have an ExclusionListing similar with the current CopyListing and we handle it while generating the sequence file. Currently we can even only define ExclusionListing's interface and provide a simple implementation just like SimpleCopyListing, and leave its extensions to separate jiras.

        Also some minor comments on the code:

        1. We can use this chance to remove the following unnecessary unboxing.
          if (mapBandwidth.intValue() <= 0) {
            throw new IllegalArgumentException("Bandwidth specified is not " +
                "positive: " + mapBandwidth);
          }
          
        2. Any reason to delete the original testParseNumListstatusThreads test?
        3. In (initializeExclusionPatterns), we can IOUtils#cleanup instead of reader.close and put the following in a "try-finally" block.
              InputStream is = new FileInputStream(new File(exclusionsPath.getName()));
              BufferedReader reader = new BufferedReader(new InputStreamReader(is,
                  Charset.forName("UTF-8")));
              String line;
              while ((line = reader.readLine()) != null) {
                exclusionPatterns.add(Pattern.compile(line));
              }
              reader.close();
          
        4. Maybe we can remove "map" from "distcp.map.exclusions.file"?
        Show
        Jing Zhao added a comment - Thanks again for working on this, Rich Haase ! The patch looks good to me overall. Some comments and thoughts: In the current patch we expect the user to only define regular expressions for exclusions. However, since the patch allows the user to define all the patterns into a file, it is possible that the user puts a long list of file names into the exclusion file (by writing some script e.g.), which can cause issue since the Mapper will compile every line into a regex pattern. Thus I guess what we can do here is to limit the total number of regex. Another option can be to have an ExclusionListing similar with the current CopyListing and we handle it while generating the sequence file. Currently we can even only define ExclusionListing's interface and provide a simple implementation just like SimpleCopyListing, and leave its extensions to separate jiras. Also some minor comments on the code: We can use this chance to remove the following unnecessary unboxing. if (mapBandwidth.intValue() <= 0) { throw new IllegalArgumentException( "Bandwidth specified is not " + "positive: " + mapBandwidth); } Any reason to delete the original testParseNumListstatusThreads test? In ( initializeExclusionPatterns ), we can IOUtils#cleanup instead of reader.close and put the following in a "try-finally" block. InputStream is = new FileInputStream( new File(exclusionsPath.getName())); BufferedReader reader = new BufferedReader( new InputStreamReader(is, Charset.forName( "UTF-8" ))); String line; while ((line = reader.readLine()) != null ) { exclusionPatterns.add(Pattern.compile(line)); } reader.close(); Maybe we can remove "map" from "distcp.map.exclusions.file"?
        Hide
        Rich Haase added a comment -

        Jing Zhao Thanks for the quick review!

        1. I'll updated the patch to skip unboxing mapBandwidth.
        2. I think that may have been caused when I rebased the patch against 3.0.0. In any case, I will be sure to fix the patch so it doesn't delete tests!
        3. Will do.
        4. Will do.

        Show
        Rich Haase added a comment - Jing Zhao Thanks for the quick review! 1. I'll updated the patch to skip unboxing mapBandwidth. 2. I think that may have been caused when I rebased the patch against 3.0.0. In any case, I will be sure to fix the patch so it doesn't delete tests! 3. Will do. 4. Will do.
        Hide
        Rich Haase added a comment -

        I'm going to review the patch again this evening and think about how an ExclusionListing interface would work. I like the idea of a more extensible interface.

        Show
        Rich Haase added a comment - I'm going to review the patch again this evening and think about how an ExclusionListing interface would work. I like the idea of a more extensible interface.
        Hide
        Rich Haase added a comment -

        Fixed issues described in Jing Zhao comments.

        Show
        Rich Haase added a comment - Fixed issues described in Jing Zhao comments.
        Hide
        Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 33s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 javac 7m 26s There were no new javac warning messages.
        +1 javadoc 9m 33s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 25s There were no new checkstyle issues.
        +1 whitespace 0m 5s The patch has no lines that end in whitespace.
        +1 install 1m 34s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 0m 40s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 tools/hadoop tests 6m 7s Tests passed in hadoop-distcp.
            41m 23s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12730946/HADOOP-1540.005.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 185e63a
        hadoop-distcp test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6509/artifact/patchprocess/testrun_hadoop-distcp.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6509/testReport/
        Java 1.7.0_55
        uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6509/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 33s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 26s There were no new javac warning messages. +1 javadoc 9m 33s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 25s There were no new checkstyle issues. +1 whitespace 0m 5s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 0m 40s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 tools/hadoop tests 6m 7s Tests passed in hadoop-distcp.     41m 23s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12730946/HADOOP-1540.005.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 185e63a hadoop-distcp test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6509/artifact/patchprocess/testrun_hadoop-distcp.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6509/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6509/console This message was automatically generated.
        Hide
        Rich Haase added a comment -

        Found a bug in the patch. The CopyCommitter doesn't know about exclusions so it will throw an exception when it tries to preserve permissions on a Path that was excluded. Need to refactor the change to separate exclusion logic into a class that is usable by the CopyMapper and CopyCommiter. Since the code has to be refactored to work correctly it makes sense to define an interface for exclusions.

        Show
        Rich Haase added a comment - Found a bug in the patch. The CopyCommitter doesn't know about exclusions so it will throw an exception when it tries to preserve permissions on a Path that was excluded. Need to refactor the change to separate exclusion logic into a class that is usable by the CopyMapper and CopyCommiter. Since the code has to be refactored to work correctly it makes sense to define an interface for exclusions.
        Hide
        Rich Haase added a comment -

        Refactored the patch to do exclusion filtering while building the CopyListing. It turns out there is a method (SimpleCopyListing#shouldCopy) which always returns true. I've added a couple of basic classes to perform the default (always true) behavior and a SimpleCopyFilter class, which uses a string compare to determine what should be excluded from the copy. I think this design will be a bit more flexible in future, and it avoids having mappers which get a chunk of files to copy that should all be excluded.

        Show
        Rich Haase added a comment - Refactored the patch to do exclusion filtering while building the CopyListing. It turns out there is a method (SimpleCopyListing#shouldCopy) which always returns true. I've added a couple of basic classes to perform the default (always true) behavior and a SimpleCopyFilter class, which uses a string compare to determine what should be excluded from the copy. I think this design will be a bit more flexible in future, and it avoids having mappers which get a chunk of files to copy that should all be excluded.
        Hide
        Zoran Dimitrijevic added a comment -

        1. hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java:564
        Minor: extra space in the comment.

        2. hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java
        Refactoring of parsing logic should have been a separate patch. This will be harder to cherry-pick to older branches. But since this is a good refactor change, and I am new to hadoop community, so it's fine with me.

        3. hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java:329
        Minor: space missing between - and 1 (-1 => - 1)

        4. hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/package-info.java
        Is this really part of this patch? Again, I am new to Hadoop community - so if it's ok to combine logically different changes it's definitely good.

        5. hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyFilter.java
        It would be more useful if it is matching glob expressions - matching substrings is a very unusual filter for file-list filtering and many users will be puzzled what to do. I would suggest if we extend this right now instead of submitting this patch as is - for example, *tmp would match filenames ending with tmp, and not any file that happens to contain tmp in it. Or in the unittest "test" filter matching /user/testing is not what I would expect.

        Otherwise, looks good to me.

        Show
        Zoran Dimitrijevic added a comment - 1. hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java:564 Minor: extra space in the comment. 2. hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java Refactoring of parsing logic should have been a separate patch. This will be harder to cherry-pick to older branches. But since this is a good refactor change, and I am new to hadoop community, so it's fine with me. 3. hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java:329 Minor: space missing between - and 1 (-1 => - 1) 4. hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/package-info.java Is this really part of this patch? Again, I am new to Hadoop community - so if it's ok to combine logically different changes it's definitely good. 5. hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyFilter.java It would be more useful if it is matching glob expressions - matching substrings is a very unusual filter for file-list filtering and many users will be puzzled what to do. I would suggest if we extend this right now instead of submitting this patch as is - for example, *tmp would match filenames ending with tmp, and not any file that happens to contain tmp in it. Or in the unittest "test" filter matching /user/testing is not what I would expect. Otherwise, looks good to me.
        Hide
        Rich Haase added a comment -

        Zoran Dimitrijevic Thanks for the comments!

        #1 and #3 I'll fix those space issues for the next rev of the patch.

        #2 and #4 were added because checkstyle failed if I didn't make those changes. I'd have preferred to leave them alone. Maybe someone can comment on how to avoid these kinds of checkstyle issues?

        #5 You are absolutely right. My initial pass at the patch used regex patterns. I switched that logic only because at the time I was doing exclusion filtering in the CopyMapper and compiling lots of regex in every mapper was likely to be expensive with large filter lists. Since we are only doing filtering while building the CopyListing it's probably not as big a deal to use regex, although I am open to alternate suggestions.

        Show
        Rich Haase added a comment - Zoran Dimitrijevic Thanks for the comments! #1 and #3 I'll fix those space issues for the next rev of the patch. #2 and #4 were added because checkstyle failed if I didn't make those changes. I'd have preferred to leave them alone. Maybe someone can comment on how to avoid these kinds of checkstyle issues? #5 You are absolutely right. My initial pass at the patch used regex patterns. I switched that logic only because at the time I was doing exclusion filtering in the CopyMapper and compiling lots of regex in every mapper was likely to be expensive with large filter lists. Since we are only doing filtering while building the CopyListing it's probably not as big a deal to use regex, although I am open to alternate suggestions.
        Hide
        Zoran Dimitrijevic added a comment -

        And one minor comment related to hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestSimpleCopyFilter.java

        Adding another test to test the case with multiple rules and making sure that all rules are applied when filtering seems like a good idea here. Two simple additional tests would be sufficient.

        Show
        Zoran Dimitrijevic added a comment - And one minor comment related to hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestSimpleCopyFilter.java Adding another test to test the case with multiple rules and making sure that all rules are applied when filtering seems like a good idea here. Two simple additional tests would be sufficient.
        Hide
        Rich Haase added a comment -

        Good idea. I'll make those changes and resubmit. Thanks again for the review!

        Show
        Rich Haase added a comment - Good idea. I'll make those changes and resubmit. Thanks again for the review!
        Hide
        Zoran Dimitrijevic added a comment -

        #5: we were experiencing performance issues for large number of files only because of RPCs to either namenode or to s3. Filtering each file name locally using a small number of compiled regex or glob rules should not be a big deal, especially since it's optional. For example, sorting a big filelist that we do now is much more expensive.

        Thank you for your patch!

        Show
        Zoran Dimitrijevic added a comment - #5: we were experiencing performance issues for large number of files only because of RPCs to either namenode or to s3. Filtering each file name locally using a small number of compiled regex or glob rules should not be a big deal, especially since it's optional. For example, sorting a big filelist that we do now is much more expensive. Thank you for your patch!
        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 39s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 5 new or modified test files.
        +1 javac 7m 32s There were no new javac warning messages.
        +1 javadoc 9m 37s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 24s There were no new checkstyle issues.
        -1 whitespace 0m 3s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 36s mvn install still works.
        +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
        +1 findbugs 0m 37s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 tools/hadoop tests 6m 10s Tests passed in hadoop-distcp.
            41m 40s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12731502/HADOOP-1540.006.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 8f7c236
        whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6549/artifact/patchprocess/whitespace.txt
        hadoop-distcp test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6549/artifact/patchprocess/testrun_hadoop-distcp.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6549/testReport/
        Java 1.7.0_55
        uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6549/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 39s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 5 new or modified test files. +1 javac 7m 32s There were no new javac warning messages. +1 javadoc 9m 37s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 24s There were no new checkstyle issues. -1 whitespace 0m 3s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. +1 findbugs 0m 37s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 tools/hadoop tests 6m 10s Tests passed in hadoop-distcp.     41m 40s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12731502/HADOOP-1540.006.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 8f7c236 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6549/artifact/patchprocess/whitespace.txt hadoop-distcp test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6549/artifact/patchprocess/testrun_hadoop-distcp.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6549/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6549/console This message was automatically generated.
        Hide
        Rich Haase added a comment -

        Incorporated suggested changes from Zoran Dimitrijevic.
        Renamed SimpleCopyFilter to RegexCopyFilter to better reflect what the class does.

        Show
        Rich Haase added a comment - Incorporated suggested changes from Zoran Dimitrijevic . Renamed SimpleCopyFilter to RegexCopyFilter to better reflect what the class does.
        Hide
        Zoran Dimitrijevic added a comment -

        please fix: "The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix."

        RegexCopyFilter constructor is currently reading from a file which is not ideal. It would be nicer if there is an init method and keep constructor only reading filename from the config. But, again, this might not be Hadoop style, and reading a file in constructor might be ok.

        Other than that, LGTM.

        Show
        Zoran Dimitrijevic added a comment - please fix: "The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix." RegexCopyFilter constructor is currently reading from a file which is not ideal. It would be nicer if there is an init method and keep constructor only reading filename from the config. But, again, this might not be Hadoop style, and reading a file in constructor might be ok. Other than that, LGTM.
        Hide
        Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 42s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 5 new or modified test files.
        +1 javac 7m 31s There were no new javac warning messages.
        +1 javadoc 9m 37s There were no new javadoc warning messages.
        +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 29s There were no new checkstyle issues.
        +1 whitespace 0m 2s The patch has no lines that end in whitespace.
        +1 install 1m 40s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 0m 38s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 tools/hadoop tests 6m 7s Tests passed in hadoop-distcp.
            41m 47s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12731619/HADOOP-1540.007.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 08f0ae4
        hadoop-distcp test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6562/artifact/patchprocess/testrun_hadoop-distcp.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6562/testReport/
        Java 1.7.0_55
        uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6562/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 42s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 5 new or modified test files. +1 javac 7m 31s There were no new javac warning messages. +1 javadoc 9m 37s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 29s There were no new checkstyle issues. +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 40s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 0m 38s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 tools/hadoop tests 6m 7s Tests passed in hadoop-distcp.     41m 47s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12731619/HADOOP-1540.007.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 08f0ae4 hadoop-distcp test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6562/artifact/patchprocess/testrun_hadoop-distcp.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6562/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6562/console This message was automatically generated.
        Hide
        Rich Haase added a comment -

        Fixed whitespace and moved file read operations for CopyFilter into an initialize method.

        Show
        Rich Haase added a comment - Fixed whitespace and moved file read operations for CopyFilter into an initialize method.
        Hide
        Zoran Dimitrijevic added a comment -

        LGTM++

        Show
        Zoran Dimitrijevic added a comment - LGTM++
        Hide
        Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 39s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 5 new or modified test files.
        +1 javac 7m 30s There were no new javac warning messages.
        +1 javadoc 9m 37s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 28s There were no new checkstyle issues.
        +1 whitespace 0m 2s The patch has no lines that end in whitespace.
        +1 install 1m 36s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 0m 39s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 tools/hadoop tests 6m 9s Tests passed in hadoop-distcp.
            41m 43s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12731986/HADOOP-1540.008.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / ea11590
        hadoop-distcp test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6592/artifact/patchprocess/testrun_hadoop-distcp.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6592/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6592/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 39s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 5 new or modified test files. +1 javac 7m 30s There were no new javac warning messages. +1 javadoc 9m 37s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 28s There were no new checkstyle issues. +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 0m 39s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 tools/hadoop tests 6m 9s Tests passed in hadoop-distcp.     41m 43s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12731986/HADOOP-1540.008.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / ea11590 hadoop-distcp test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6592/artifact/patchprocess/testrun_hadoop-distcp.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6592/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6592/console This message was automatically generated.
        Hide
        Rich Haase added a comment -

        Jing Zhao, would you please review the latest revision of this patch when you have time? Thanks!

        Show
        Rich Haase added a comment - Jing Zhao , would you please review the latest revision of this patch when you have time? Thanks!
        Hide
        Jing Zhao added a comment -

        Sorry for the delay, Rich Haase. I will review it asap.

        Show
        Jing Zhao added a comment - Sorry for the delay, Rich Haase . I will review it asap.
        Hide
        Jing Zhao added a comment -

        The patch looks pretty good to me. Just some minors:

        1. In RegexCopyFilter and TrueCopyFilter, please make sure "@Override" annotation is added for overrided methods. Besides, "inheritDoc" is actually unnecessary unless there is extra javadoc added.
        2. RegexCopyFilter's constructor can take the file name as the parameter instead of the Configuration object.
        3. RegexCopyFilter's default constructor can be removed. For test we can pass a string to its constructor.
        4. The parameter "options" can removed from SimpleCopyListing#writeToFileListing.
        Show
        Jing Zhao added a comment - The patch looks pretty good to me. Just some minors: In RegexCopyFilter and TrueCopyFilter , please make sure "@Override" annotation is added for overrided methods. Besides, "inheritDoc" is actually unnecessary unless there is extra javadoc added. RegexCopyFilter 's constructor can take the file name as the parameter instead of the Configuration object. RegexCopyFilter 's default constructor can be removed. For test we can pass a string to its constructor. The parameter "options" can removed from SimpleCopyListing#writeToFileListing .
        Hide
        Rich Haase added a comment -

        Jing Zhao I've updated the patch with the fixed you suggested. Thanks, as always, for your help!!

        Show
        Rich Haase added a comment - Jing Zhao I've updated the patch with the fixed you suggested. Thanks, as always, for your help!!
        Hide
        Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 39s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 5 new or modified test files.
        +1 javac 7m 31s There were no new javac warning messages.
        +1 javadoc 9m 35s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 25s There were no new checkstyle issues.
        +1 whitespace 0m 2s The patch has no lines that end in whitespace.
        +1 install 1m 32s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        +1 findbugs 0m 38s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 tools/hadoop tests 6m 7s Tests passed in hadoop-distcp.
            41m 31s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12733587/HADOOP-1540.009.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 060c84e
        hadoop-distcp test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6731/artifact/patchprocess/testrun_hadoop-distcp.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6731/testReport/
        Java 1.7.0_55
        uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6731/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 39s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 5 new or modified test files. +1 javac 7m 31s There were no new javac warning messages. +1 javadoc 9m 35s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 25s There were no new checkstyle issues. +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 0m 38s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 tools/hadoop tests 6m 7s Tests passed in hadoop-distcp.     41m 31s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12733587/HADOOP-1540.009.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 060c84e hadoop-distcp test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6731/artifact/patchprocess/testrun_hadoop-distcp.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6731/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6731/console This message was automatically generated.
        Hide
        Jing Zhao added a comment - - edited

        The latest patch looks good to me. +1. I will commit it shortly.

        Show
        Jing Zhao added a comment - - edited The latest patch looks good to me. +1. I will commit it shortly.
        Hide
        Rich Haase added a comment -

        Thanks again Jing Zhao and Zoran Dimitrijevic for the reviews!

        Show
        Rich Haase added a comment - Thanks again Jing Zhao and Zoran Dimitrijevic for the reviews!
        Hide
        Jing Zhao added a comment -

        I've committed this to trunk and branch-2. Thanks Rich Haase for the contribution! And thanks Zoran Dimitrijevic for the review!

        Show
        Jing Zhao added a comment - I've committed this to trunk and branch-2. Thanks Rich Haase for the contribution! And thanks Zoran Dimitrijevic for the review!
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #7857 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7857/)
        HADOOP-1540. Support file exclusion list in distcp. Contributed by Rich Haase. (jing9: rev 0790275f058b0cf41780ad337c9150a1e8ebebc6)

        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestTrueCopyFilter.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/package-info.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyFilter.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestIntegration.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/TrueCopyFilter.java
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestRegexCopyFilter.java
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/RegexCopyFilter.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7857 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7857/ ) HADOOP-1540 . Support file exclusion list in distcp. Contributed by Rich Haase. (jing9: rev 0790275f058b0cf41780ad337c9150a1e8ebebc6) hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestTrueCopyFilter.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/package-info.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyFilter.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestIntegration.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/TrueCopyFilter.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestRegexCopyFilter.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/RegexCopyFilter.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #932 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/932/)
        HADOOP-1540. Support file exclusion list in distcp. Contributed by Rich Haase. (jing9: rev 0790275f058b0cf41780ad337c9150a1e8ebebc6)

        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestRegexCopyFilter.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/TrueCopyFilter.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/package-info.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/RegexCopyFilter.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestIntegration.java
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyFilter.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestTrueCopyFilter.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #932 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/932/ ) HADOOP-1540 . Support file exclusion list in distcp. Contributed by Rich Haase. (jing9: rev 0790275f058b0cf41780ad337c9150a1e8ebebc6) hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestRegexCopyFilter.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/TrueCopyFilter.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/package-info.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/RegexCopyFilter.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestIntegration.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyFilter.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestTrueCopyFilter.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #201 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/201/)
        HADOOP-1540. Support file exclusion list in distcp. Contributed by Rich Haase. (jing9: rev 0790275f058b0cf41780ad337c9150a1e8ebebc6)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestIntegration.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/RegexCopyFilter.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/package-info.java
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestTrueCopyFilter.java
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestRegexCopyFilter.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyFilter.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/TrueCopyFilter.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #201 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/201/ ) HADOOP-1540 . Support file exclusion list in distcp. Contributed by Rich Haase. (jing9: rev 0790275f058b0cf41780ad337c9150a1e8ebebc6) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestIntegration.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/RegexCopyFilter.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/package-info.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestTrueCopyFilter.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestRegexCopyFilter.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyFilter.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/TrueCopyFilter.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2130 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2130/)
        HADOOP-1540. Support file exclusion list in distcp. Contributed by Rich Haase. (jing9: rev 0790275f058b0cf41780ad337c9150a1e8ebebc6)

        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestRegexCopyFilter.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/TrueCopyFilter.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/RegexCopyFilter.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/package-info.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestIntegration.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyFilter.java
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestTrueCopyFilter.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2130 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2130/ ) HADOOP-1540 . Support file exclusion list in distcp. Contributed by Rich Haase. (jing9: rev 0790275f058b0cf41780ad337c9150a1e8ebebc6) hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestRegexCopyFilter.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/TrueCopyFilter.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/RegexCopyFilter.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/package-info.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestIntegration.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyFilter.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestTrueCopyFilter.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #190 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/190/)
        HADOOP-1540. Support file exclusion list in distcp. Contributed by Rich Haase. (jing9: rev 0790275f058b0cf41780ad337c9150a1e8ebebc6)

        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/RegexCopyFilter.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyFilter.java
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestIntegration.java
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestTrueCopyFilter.java
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestRegexCopyFilter.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/TrueCopyFilter.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/package-info.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #190 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/190/ ) HADOOP-1540 . Support file exclusion list in distcp. Contributed by Rich Haase. (jing9: rev 0790275f058b0cf41780ad337c9150a1e8ebebc6) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/RegexCopyFilter.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyFilter.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestIntegration.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestTrueCopyFilter.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestRegexCopyFilter.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/TrueCopyFilter.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/package-info.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #200 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/200/)
        HADOOP-1540. Support file exclusion list in distcp. Contributed by Rich Haase. (jing9: rev 0790275f058b0cf41780ad337c9150a1e8ebebc6)

        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/TrueCopyFilter.java
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/RegexCopyFilter.java
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyFilter.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestTrueCopyFilter.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/package-info.java
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestIntegration.java
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestRegexCopyFilter.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #200 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/200/ ) HADOOP-1540 . Support file exclusion list in distcp. Contributed by Rich Haase. (jing9: rev 0790275f058b0cf41780ad337c9150a1e8ebebc6) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/TrueCopyFilter.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/RegexCopyFilter.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyFilter.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestTrueCopyFilter.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/package-info.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestIntegration.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestRegexCopyFilter.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2148 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2148/)
        HADOOP-1540. Support file exclusion list in distcp. Contributed by Rich Haase. (jing9: rev 0790275f058b0cf41780ad337c9150a1e8ebebc6)

        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyFilter.java
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestTrueCopyFilter.java
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestRegexCopyFilter.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestIntegration.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/RegexCopyFilter.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/package-info.java
        • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/TrueCopyFilter.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java
        • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2148 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2148/ ) HADOOP-1540 . Support file exclusion list in distcp. Contributed by Rich Haase. (jing9: rev 0790275f058b0cf41780ad337c9150a1e8ebebc6) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyFilter.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestTrueCopyFilter.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestRegexCopyFilter.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestIntegration.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/RegexCopyFilter.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/package-info.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/TrueCopyFilter.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java

          People

          • Assignee:
            Rich Haase
            Reporter:
            Senthil Subramanian
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development