Hadoop Common
  1. Hadoop Common
  2. HADOOP-8775

MR2 distcp permits non-positive value to -bandwidth option which causes job never to complete

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha, 0.23.5
    • Component/s: None
    • Labels:
      None

      Description

      The likelihood that someone would want to enter a non-positive value for -bandwidth seems really low. However, the job would never complete if a non-positive value was specified. It'd just get stuck at map 100%. Luckily, a positive value would always lead to the job completing.

      bash-4.1$ hadoop distcp -bandwidth 0 hdfs://c1204.hal.cloudera.com:17020/user/hdfs/in-dir hdfs://c1204.hal.cloudera.com:17020/user/hdfs/in-dir58
      hadoop distcp -bandwidth 0 hdfs://c1204.hal.cloudera.com:17020/user/hdfs/in-dir hdfs://c1204.hal.cloudera.com:17020/user/hdfs/in-dir58
      12/05/23 15:53:01 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, maxMaps=20, sslConfigurationFile='null', copyStrategy='uniformsiz\
      e', sourceFileListing=null, sourcePaths=[hdfs://c1204.hal.cloudera.com:17020/user/hdfs/in-dir], targetPath=hdfs://c1204.hal.cloudera.com:17020/user/hdfs/in-dir58}
      12/05/23 15:53:02 WARN conf.Configuration: io.sort.mb is deprecated. Instead, use mapreduce.task.io.sort.mb
      12/05/23 15:53:02 WARN conf.Configuration: io.sort.factor is deprecated. Instead, use mapreduce.task.io.sort.factor
      12/05/23 15:53:02 INFO util.NativeCodeLoader: Loaded the native-hadoop library
      12/05/23 15:53:03 INFO mapreduce.JobSubmitter: number of splits:3
      12/05/23 15:53:04 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar
      12/05/23 15:53:04 WARN conf.Configuration: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
      12/05/23 15:53:04 WARN conf.Configuration: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
      12/05/23 15:53:04 WARN conf.Configuration: mapred.mapoutput.value.class is deprecated. Instead, use mapreduce.map.output.value.class
      12/05/23 15:53:04 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
      12/05/23 15:53:04 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name
      12/05/23 15:53:04 WARN conf.Configuration: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
      12/05/23 15:53:04 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
      12/05/23 15:53:04 WARN conf.Configuration: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
      12/05/23 15:53:04 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
      12/05/23 15:53:04 WARN conf.Configuration: mapred.mapoutput.key.class is deprecated. Instead, use mapreduce.map.output.key.class
      12/05/23 15:53:04 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
      12/05/23 15:53:04 INFO mapred.ResourceMgrDelegate: Submitted application application_1337808305464_0014 to ResourceManager at c1204.hal.cloudera.com/172.29.98.195:8040
      12/05/23 15:53:04 INFO mapreduce.Job: The url to track the job: http://auto0:8088/proxy/application_1337808305464_0014/
      12/05/23 15:53:04 INFO tools.DistCp: DistCp job-id: job_1337808305464_0014
      12/05/23 15:53:04 INFO mapreduce.Job: Running job: job_1337808305464_0014
      12/05/23 15:53:09 INFO mapreduce.Job: Job job_1337808305464_0014 running in uber mode : false
      12/05/23 15:53:09 INFO mapreduce.Job:  map 0% reduce 0%
      12/05/23 15:53:14 INFO mapreduce.Job:  map 33% reduce 0%
      12/05/23 15:53:19 INFO mapreduce.Job:  map 100% reduce 0%
      
      1. HADOOP-8775.patch
        2 kB
        Sandy Ryza
      2. diff2.txt
        1 kB
        Sandy Ryza

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #392 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/392/)
        svn merge -c 1382119 FIXES: HADOOP-8775. MR2 distcp permits non-positive value to -bandwidth option which causes job never to complete. Contributed by Sandy Ryza. (Revision 1392519)

        Result = UNSTABLE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1392519
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java
        • /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #392 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/392/ ) svn merge -c 1382119 FIXES: HADOOP-8775 . MR2 distcp permits non-positive value to -bandwidth option which causes job never to complete. Contributed by Sandy Ryza. (Revision 1392519) Result = UNSTABLE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1392519 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
        Hide
        Robert Joseph Evans added a comment -

        I pulled this into branch-0.23

        Show
        Robert Joseph Evans added a comment - I pulled this into branch-0.23
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1190 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1190/)
        HADOOP-8775. MR2 distcp permits non-positive value to -bandwidth option which causes job never to complete. Contributed by Sandy Ryza. (Revision 1382119)

        Result = SUCCESS
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382119
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1190 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1190/ ) HADOOP-8775 . MR2 distcp permits non-positive value to -bandwidth option which causes job never to complete. Contributed by Sandy Ryza. (Revision 1382119) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382119 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1159 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1159/)
        HADOOP-8775. MR2 distcp permits non-positive value to -bandwidth option which causes job never to complete. Contributed by Sandy Ryza. (Revision 1382119)

        Result = FAILURE
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382119
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1159 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1159/ ) HADOOP-8775 . MR2 distcp permits non-positive value to -bandwidth option which causes job never to complete. Contributed by Sandy Ryza. (Revision 1382119) Result = FAILURE atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382119 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2732 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2732/)
        HADOOP-8775. MR2 distcp permits non-positive value to -bandwidth option which causes job never to complete. Contributed by Sandy Ryza. (Revision 1382119)

        Result = FAILURE
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382119
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2732 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2732/ ) HADOOP-8775 . MR2 distcp permits non-positive value to -bandwidth option which causes job never to complete. Contributed by Sandy Ryza. (Revision 1382119) Result = FAILURE atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382119 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2771 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2771/)
        HADOOP-8775. MR2 distcp permits non-positive value to -bandwidth option which causes job never to complete. Contributed by Sandy Ryza. (Revision 1382119)

        Result = SUCCESS
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382119
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2771 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2771/ ) HADOOP-8775 . MR2 distcp permits non-positive value to -bandwidth option which causes job never to complete. Contributed by Sandy Ryza. (Revision 1382119) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382119 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2708 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2708/)
        HADOOP-8775. MR2 distcp permits non-positive value to -bandwidth option which causes job never to complete. Contributed by Sandy Ryza. (Revision 1382119)

        Result = SUCCESS
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382119
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2708 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2708/ ) HADOOP-8775 . MR2 distcp permits non-positive value to -bandwidth option which causes job never to complete. Contributed by Sandy Ryza. (Revision 1382119) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382119 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
        Hide
        Aaron T. Myers added a comment -

        I've just committed this to trunk and branch-2. Thanks a lot for the contribution, Sandy.

        Show
        Aaron T. Myers added a comment - I've just committed this to trunk and branch-2. Thanks a lot for the contribution, Sandy.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12544146/HADOOP-8775.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-distcp.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1421//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1421//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12544146/HADOOP-8775.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-distcp. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1421//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1421//console This message is automatically generated.
        Hide
        Aaron T. Myers added a comment -

        Thanks a lot for addressing my comments, Sandy. The latest patch looks good to me. +1 pending a clean Jenkins report.

        Show
        Aaron T. Myers added a comment - Thanks a lot for addressing my comments, Sandy. The latest patch looks good to me. +1 pending a clean Jenkins report.
        Hide
        Aaron T. Myers added a comment -

        The patch looks pretty good to me, but it should be pretty easy to add a little test case for this. See TestOptionsParser for some examples. There's even one in there which tests the bandwidth option.

        Also, please don't set the "fix versions" field until the JIRA is actually committed. Use the "target versions" field to express what versions you'd like to see this JIRA committed to.

        Also, per the how to contribute wiki, please name patch files using the form "HADOOP-8775.patch" instead of "diff2.txt".

        Show
        Aaron T. Myers added a comment - The patch looks pretty good to me, but it should be pretty easy to add a little test case for this. See TestOptionsParser for some examples. There's even one in there which tests the bandwidth option. Also, please don't set the "fix versions" field until the JIRA is actually committed. Use the "target versions" field to express what versions you'd like to see this JIRA committed to. Also, per the how to contribute wiki , please name patch files using the form " HADOOP-8775 .patch" instead of "diff2.txt".
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12544100/diff2.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-distcp.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1417//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1417//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12544100/diff2.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-distcp. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1417//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1417//console This message is automatically generated.
        Hide
        Sandy Ryza added a comment -

        I made it throw an IllegalArgumentException, as it would do if the bandwidth entered were not an integer.

        Show
        Sandy Ryza added a comment - I made it throw an IllegalArgumentException, as it would do if the bandwidth entered were not an integer.

          People

          • Assignee:
            Sandy Ryza
            Reporter:
            Sandy Ryza
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development