Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-664

distcp with -delete option does not display number of files deleted from the target that were not present on source

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.1
    • Fix Version/s: 0.21.0
    • Component/s: distcp
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      distcp with -delete option should provide information on total number of files deleted from the target that were not present on the source.

      1. d_deletedPathsCount.patch
        2 kB
        Ravi Gummadi
      2. d_deletedPathsCount664.patch
        2 kB
        Ravi Gummadi

        Activity

        Hide
        Ravi Gummadi added a comment -

        Attaching patch for trunk for displaying the number of paths(files + directories) deleted with -delete.
        Please review and provide your comments.

        Show
        Ravi Gummadi added a comment - Attaching patch for trunk for displaying the number of paths(files + directories) deleted with -delete. Please review and provide your comments.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12411897/d_deletedPathsCount.patch
        against trunk revision 808320.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/526/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/526/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/526/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/526/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12411897/d_deletedPathsCount.patch against trunk revision 808320. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/526/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/526/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/526/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/526/console This message is automatically generated.
        Hide
        Ravi Gummadi added a comment -

        Failed test is not related to this patch.

        I did not add a testcase with this patch as this patch just makes distcp to display the number of paths deleted with -delete option. Tested manually with -delete option.

        Show
        Ravi Gummadi added a comment - Failed test is not related to this patch. I did not add a testcase with this patch as this patch just makes distcp to display the number of paths deleted with -delete option. Tested manually with -delete option.
        Hide
        Ravi Gummadi added a comment -

        Attaching patch that applies after MAPREDUCE-649, MAPREDUCE-654 and MAPREDUCE-645 are committed.

        Please review and provide your comments.

        Show
        Ravi Gummadi added a comment - Attaching patch that applies after MAPREDUCE-649 , MAPREDUCE-654 and MAPREDUCE-645 are committed. Please review and provide your comments.
        Hide
        Chris Douglas added a comment -
             [exec] +1 overall.  
             [exec] 
             [exec]     +1 @author.  The patch does not contain any @author tags.
             [exec] 
             [exec]     -1 tests included.  The patch doesn't appear to include any new or modified tests.
             [exec]                         Please justify why no new tests are needed for this patch.
             [exec]                         Also please list what manual steps were performed to verify this patch.
             [exec] 
             [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
             [exec] 
             [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
             [exec] 
             [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
             [exec] 
             [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
        

        TestCopyFiles passed; this affects no other components. It requires no tests as it adds a diagnostic message. Verified the correctness of the message by inspecting code and testcase output.

        Show
        Chris Douglas added a comment - [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no new tests are needed for this patch. [exec] Also please list what manual steps were performed to verify this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. TestCopyFiles passed; this affects no other components. It requires no tests as it adds a diagnostic message. Verified the correctness of the message by inspecting code and testcase output.
        Hide
        Chris Douglas added a comment -

        +1

        I committed this. Thanks, Ravi!

        Show
        Chris Douglas added a comment - +1 I committed this. Thanks, Ravi!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #49 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/49/)
        . Display the number of files deleted by distcp when the
        -delete option is specified. Contributed by Ravi Gummadi

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #49 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/49/ ) . Display the number of files deleted by distcp when the -delete option is specified. Contributed by Ravi Gummadi
        Hide
        Nigel Daley added a comment -

        Same problem. No automated unit test when there really could have been. How to you ensure this never breaks?

        Show
        Nigel Daley added a comment - Same problem. No automated unit test when there really could have been. How to you ensure this never breaks?
        Hide
        Chris Douglas added a comment -

        No automated unit test when there really could have been. How to you ensure this never breaks?

        Good point. A test for this should be included in HADOOP-1008

        Show
        Chris Douglas added a comment - No automated unit test when there really could have been. How to you ensure this never breaks? Good point. A test for this should be included in HADOOP-1008
        Hide
        gary murry added a comment -

        I assume you meant MAPREDUCE-1008. Not HADOOP-1008.

        Show
        gary murry added a comment - I assume you meant MAPREDUCE-1008 . Not HADOOP-1008 .
        Hide
        Chris Douglas added a comment -

        I assume you meant MAPREDUCE-1008. Not HADOOP-1008.

        Yes; thanks for the correction

        Show
        Chris Douglas added a comment - I assume you meant MAPREDUCE-1008 . Not HADOOP-1008 . Yes; thanks for the correction

          People

          • Assignee:
            Ravi Gummadi
            Reporter:
            Suhas Gogate
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development