Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-8775

MR2 distcp permits non-positive value to -bandwidth option which causes job never to complete

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.0.0-alpha
    • 2.0.2-alpha, 0.23.5
    • None
    • None

    Description

      The likelihood that someone would want to enter a non-positive value for -bandwidth seems really low. However, the job would never complete if a non-positive value was specified. It'd just get stuck at map 100%. Luckily, a positive value would always lead to the job completing.

      bash-4.1$ hadoop distcp -bandwidth 0 hdfs://c1204.hal.cloudera.com:17020/user/hdfs/in-dir hdfs://c1204.hal.cloudera.com:17020/user/hdfs/in-dir58
      hadoop distcp -bandwidth 0 hdfs://c1204.hal.cloudera.com:17020/user/hdfs/in-dir hdfs://c1204.hal.cloudera.com:17020/user/hdfs/in-dir58
      12/05/23 15:53:01 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, maxMaps=20, sslConfigurationFile='null', copyStrategy='uniformsiz\
      e', sourceFileListing=null, sourcePaths=[hdfs://c1204.hal.cloudera.com:17020/user/hdfs/in-dir], targetPath=hdfs://c1204.hal.cloudera.com:17020/user/hdfs/in-dir58}
      12/05/23 15:53:02 WARN conf.Configuration: io.sort.mb is deprecated. Instead, use mapreduce.task.io.sort.mb
      12/05/23 15:53:02 WARN conf.Configuration: io.sort.factor is deprecated. Instead, use mapreduce.task.io.sort.factor
      12/05/23 15:53:02 INFO util.NativeCodeLoader: Loaded the native-hadoop library
      12/05/23 15:53:03 INFO mapreduce.JobSubmitter: number of splits:3
      12/05/23 15:53:04 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar
      12/05/23 15:53:04 WARN conf.Configuration: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
      12/05/23 15:53:04 WARN conf.Configuration: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
      12/05/23 15:53:04 WARN conf.Configuration: mapred.mapoutput.value.class is deprecated. Instead, use mapreduce.map.output.value.class
      12/05/23 15:53:04 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
      12/05/23 15:53:04 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name
      12/05/23 15:53:04 WARN conf.Configuration: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
      12/05/23 15:53:04 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
      12/05/23 15:53:04 WARN conf.Configuration: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
      12/05/23 15:53:04 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
      12/05/23 15:53:04 WARN conf.Configuration: mapred.mapoutput.key.class is deprecated. Instead, use mapreduce.map.output.key.class
      12/05/23 15:53:04 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
      12/05/23 15:53:04 INFO mapred.ResourceMgrDelegate: Submitted application application_1337808305464_0014 to ResourceManager at c1204.hal.cloudera.com/172.29.98.195:8040
      12/05/23 15:53:04 INFO mapreduce.Job: The url to track the job: http://auto0:8088/proxy/application_1337808305464_0014/
      12/05/23 15:53:04 INFO tools.DistCp: DistCp job-id: job_1337808305464_0014
      12/05/23 15:53:04 INFO mapreduce.Job: Running job: job_1337808305464_0014
      12/05/23 15:53:09 INFO mapreduce.Job: Job job_1337808305464_0014 running in uber mode : false
      12/05/23 15:53:09 INFO mapreduce.Job:  map 0% reduce 0%
      12/05/23 15:53:14 INFO mapreduce.Job:  map 33% reduce 0%
      12/05/23 15:53:19 INFO mapreduce.Job:  map 100% reduce 0%
      

      Attachments

        1. diff2.txt
          1 kB
          Sandy Ryza
        2. HADOOP-8775.patch
          2 kB
          Sandy Ryza

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            sandyr Sandy Ryza
            sandyr Sandy Ryza
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment