Hadoop Common
  1. Hadoop Common
  2. HADOOP-313

A stand alone driver for individual tasks

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.5.0
    • Component/s: None
    • Labels:
      None

      Description

      This is a tool to reproduce problems and to run unit tests involving either a map or reduce task.
      You just give it a reduce directory on the command line.

      Usage: java org.apache.hadoop.mapred.StandaloneReduceTask <taskdir> [<limitmaps>]
      taskdir name encodes: task_<jobid>r<partition>_<attempt>
      taskdir contains job.xml and one or more input files named: map_<dddd>.out
      You should run with the same -Xmx option as the TaskTracker child JVM

      1. stand-alone-task.patch
        18 kB
        Owen O'Malley
      2. stand-alone-task.patch
        11 kB
        Owen O'Malley
      3. sareduce.patch
        10 kB
        Michel Tourn

        Issue Links

          Activity

          Hide
          Doug Cutting added a comment -

          Shouldn't this go into the "test" tree rather than the core?

          Show
          Doug Cutting added a comment - Shouldn't this go into the "test" tree rather than the core?
          Hide
          Owen O'Malley added a comment -

          Actually, it isn't for testing at all. It is for debugging a reduce crash in place on the production machines. Michel manually caught the reduce inputs, but in general we also need to add a configuration setting that lets a job request that some reduce task's inputs be kept. Maybe something like:
          mapred.keep.reduce.inputs=...
          Where the legal values are "all", "none", or a list of integers of which reduce tasks to keep the inputs for.

          Show
          Owen O'Malley added a comment - Actually, it isn't for testing at all. It is for debugging a reduce crash in place on the production machines. Michel manually caught the reduce inputs, but in general we also need to add a configuration setting that lets a job request that some reduce task's inputs be kept. Maybe something like: mapred.keep.reduce.inputs=... Where the legal values are "all", "none", or a list of integers of which reduce tasks to keep the inputs for.
          Hide
          eric baldeschwieler added a comment -

          Why not an option to keep inputs for failed reduces?
          Also recall HADOOP-91.

          Show
          eric baldeschwieler added a comment - Why not an option to keep inputs for failed reduces? Also recall HADOOP-91 .
          Hide
          Owen O'Malley added a comment -

          An option to keep failed reduces would be useful, but I've certainly seen cases before where I wanted to run a particular fragment in the debugger, even if it didn't crash.

          Show
          Owen O'Malley added a comment - An option to keep failed reduces would be useful, but I've certainly seen cases before where I wanted to run a particular fragment in the debugger, even if it didn't crash.
          Hide
          Owen O'Malley added a comment -

          This patch does a lot more:
          1. It moves the jobId and partition fields from ReduceTask up into Task.
          2. It replaces Michel's stand-alone reduce runner, with a more general map or reduce runner.
          3. It adds new task localizations into the job.xml.
          For all tasks:
          a. map.task.id = the task id
          b. mapred.task.is.map = is this a map
          c. mapred.task.partition = numeric task id
          d. mapred.job.id = job id
          For maps:
          a. map.input.file = the file that we are reading
          b. map.input.start = the offset in the file to start at
          c. map.input.length = the number of bytes in the split
          For reduces:
          a. mapred.map.tasks = correct number of maps
          These new attributes allow me to reconstruct the MapTask or ReduceTask with just the job.xml.
          4. A new configuration variable keep.failed.task.files that tells the system to keep files/directories for tasks that fail. This attribute can be set on a particular JobConf.

          Show
          Owen O'Malley added a comment - This patch does a lot more: 1. It moves the jobId and partition fields from ReduceTask up into Task. 2. It replaces Michel's stand-alone reduce runner, with a more general map or reduce runner. 3. It adds new task localizations into the job.xml. For all tasks: a. map.task.id = the task id b. mapred.task.is.map = is this a map c. mapred.task.partition = numeric task id d. mapred.job.id = job id For maps: a. map.input.file = the file that we are reading b. map.input.start = the offset in the file to start at c. map.input.length = the number of bytes in the split For reduces: a. mapred.map.tasks = correct number of maps These new attributes allow me to reconstruct the MapTask or ReduceTask with just the job.xml. 4. A new configuration variable keep.failed.task.files that tells the system to keep files/directories for tasks that fail. This attribute can be set on a particular JobConf.
          Hide
          Owen O'Malley added a comment -

          One more comment, the usage is:

          bin/hadoop org.apache.hadoop.mapred.IsolationRunner <local-dir>/taskTracker/<taskid>/job.xml

          Show
          Owen O'Malley added a comment - One more comment, the usage is: bin/hadoop org.apache.hadoop.mapred.IsolationRunner <local-dir>/taskTracker/<taskid>/job.xml
          Hide
          Doug Cutting added a comment -

          This looks good, except I think you forgot to add IsolationRunner.java.

          Show
          Doug Cutting added a comment - This looks good, except I think you forgot to add IsolationRunner.java.
          Hide
          Owen O'Malley added a comment -

          Here is the updated patch that includes the IsolationRunner.

          Show
          Owen O'Malley added a comment - Here is the updated patch that includes the IsolationRunner.
          Hide
          Doug Cutting added a comment -

          I just committed this. Thanks, Owen.

          Show
          Doug Cutting added a comment - I just committed this. Thanks, Owen.

            People

            • Assignee:
              Michel Tourn
              Reporter:
              Michel Tourn
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development