Hadoop Common
  1. Hadoop Common
  2. HADOOP-5675

DistCp should not launch a job if it is not necessary

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Distcp will no longer start jobs that move no data.

      Description

      Currently, DistCp launch a job even if it is not necessary. In such case, the job does nothing.

      1. 5675_20090428.patch
        2 kB
        Tsz Wo Nicholas Sze
      2. 5675_20090422.patch
        2 kB
        Tsz Wo Nicholas Sze
      3. 5675_20090421.patch
        2 kB
        Tsz Wo Nicholas Sze
      4. 5675_20090414.patch
        2 kB
        Tsz Wo Nicholas Sze

        Activity

        Hide
        Tsz Wo Nicholas Sze added a comment -

        5675_20090414.patch: not starting a job if there is no bytes.

        Show
        Tsz Wo Nicholas Sze added a comment - 5675_20090414.patch: not starting a job if there is no bytes.
        Hide
        Chris Douglas added a comment -

        +1

        Show
        Chris Douglas added a comment - +1
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12405477/5675_20090414.patch
        against trunk revision 765815.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no tests are needed for this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/204/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/204/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/204/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/204/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12405477/5675_20090414.patch against trunk revision 765815. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/204/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/204/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/204/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/204/console This message is automatically generated.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        If there are only zero length files, we have byteCount == 0. However, DistCp should launch a job to copy these zero length files in this case. Instead of byteCount, we should use fileCount, which counts how many files needed to be copied.

        Edited summary and description to reflect this.

        Show
        Tsz Wo Nicholas Sze added a comment - If there are only zero length files, we have byteCount == 0. However, DistCp should launch a job to copy these zero length files in this case. Instead of byteCount, we should use fileCount, which counts how many files needed to be copied. Edited summary and description to reflect this.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        5675_20090421.patch: use fileCount

        Show
        Tsz Wo Nicholas Sze added a comment - 5675_20090421.patch: use fileCount
        Hide
        Suhas Gogate added a comment -

        Tested the patch and here are some comments w.r.t. following output I see,

        1. suggestions for counter names: srcCount => sourceFilesCount, fileCount => filesToCopyCount, byteCount => bytesToCopyCount

        2. Remove white spaces between counter name and value.

        09/04/21 23:37:30 INFO tools.DistCp: srcCount = 6
        09/04/21 23:37:30 INFO tools.DistCp: fileCount= 2
        09/04/21 23:37:30 INFO tools.DistCp: byteCount= 341.5k

        Show
        Suhas Gogate added a comment - Tested the patch and here are some comments w.r.t. following output I see, 1. suggestions for counter names: srcCount => sourceFilesCount, fileCount => filesToCopyCount, byteCount => bytesToCopyCount 2. Remove white spaces between counter name and value. 09/04/21 23:37:30 INFO tools.DistCp: srcCount = 6 09/04/21 23:37:30 INFO tools.DistCp: fileCount= 2 09/04/21 23:37:30 INFO tools.DistCp: byteCount= 341.5k
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Suhas, thanks for your comment.

        5675_20090422.patch: changed the log messages.

        Show
        Tsz Wo Nicholas Sze added a comment - Suhas, thanks for your comment. 5675_20090422.patch: changed the log messages.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        5675_20090428.patch: renamed "sourceFilesCount" to "sourcePathsCount" since the count also includes directories.

        Show
        Tsz Wo Nicholas Sze added a comment - 5675_20090428.patch: renamed "sourceFilesCount" to "sourcePathsCount" since the count also includes directories.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12406687/5675_20090428.patch
        against trunk revision 769923.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no tests are needed for this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/262/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/262/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/262/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/262/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12406687/5675_20090428.patch against trunk revision 769923. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/262/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/262/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/262/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/262/console This message is automatically generated.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Tested manually. No new tests added.

        Show
        Tsz Wo Nicholas Sze added a comment - Tested manually. No new tests added.
        Hide
        Chris Douglas added a comment -

        +1

        Show
        Chris Douglas added a comment - +1
        Hide
        Chris Douglas added a comment -

        I committed this. Thanks, Nicholas

        Show
        Chris Douglas added a comment - I committed this. Thanks, Nicholas
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk #827 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/827/)
        . Do not launch a job if DistCp has no work to do. Contributed by Tsz Wo (Nicholas), SZE

        Show
        Hudson added a comment - Integrated in Hadoop-trunk #827 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/827/ ) . Do not launch a job if DistCp has no work to do. Contributed by Tsz Wo (Nicholas), SZE
        Hide
        Chris Douglas added a comment -

        I committed the wrong patch.

        Show
        Chris Douglas added a comment - I committed the wrong patch.
        Hide
        Chris Douglas added a comment -

        I (re)committed this.

        Show
        Chris Douglas added a comment - I (re)committed this.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk #834 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/834/)
        . Do not launch a job if DistCp has no work to do. Contributed by Tsz Wo (Nicholas), SZE
        Revert

        Show
        Hudson added a comment - Integrated in Hadoop-trunk #834 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/834/ ) . Do not launch a job if DistCp has no work to do. Contributed by Tsz Wo (Nicholas), SZE Revert
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Tested 5675_20090428.patch with 0.20: it applies and passed TestCopyFiles, which is the only related test.

        Show
        Tsz Wo Nicholas Sze added a comment - Tested 5675_20090428.patch with 0.20: it applies and passed TestCopyFiles, which is the only related test.

          People

          • Assignee:
            Tsz Wo Nicholas Sze
            Reporter:
            Tsz Wo Nicholas Sze
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development