[MAPREDUCE-1853] MultipleOutputs does not cache TaskAttemptContext

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.21.0, 0.22.0
• Fix Version/s: 0.21.0, 0.22.0
• Component/s: task

Environment

OSX 10.6
java6

• Hadoop Flags: Reviewed
• Assignee: Torsten Curdt
• Reporter: Torsten Curdt
• Created: 10/Jun/10 14:35
• Updated: 23/Nov/11 06:03
• Resolved: 16/Jun/10 11:25

Description

In MultipleOutputs there is

 private TaskAttemptContext getContext(String nameOutput) throws IOException {
    // The following trick leverages the instantiation of a record writer via
    // the job thus supporting arbitrary output formats.
    Job job = new Job(context.getConfiguration());
    job.setOutputFormatClass(getNamedOutputFormatClass(context, nameOutput));
    job.setOutputKeyClass(getNamedOutputKeyClass(context, nameOutput));
    job.setOutputValueClass(getNamedOutputValueClass(context, nameOutput));
    TaskAttemptContext taskContext = 
      new TaskAttemptContextImpl(job.getConfiguration(), 
                                 context.getTaskAttemptID());
    return taskContext;
  }

so for every reduce call it creates a new Job instance ...which creates a new LocalJobRunner.
That does not sound like a good idea.

You end up with a flood of "jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized"

This should probably also be added to 0.22.

Issue Links

• is related to: MAPREDUCE-2740 MultipleOutputs in new API creates needless TaskAttemptContexts