[MAPREDUCE-5482] Release kvoffsets and kvindices before going to merge phase

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.2.0
• Component/s: task
• Labels: patch

Environment

Any Environment

• Target Version/s: 1.2.1
• Assignee: Unassigned
• Reporter: Lijie Xu
• Created: 27/Aug/13 02:38
• Updated: 27/Aug/13 02:38

Description

In org.apache.hadoop.mapred.MapTask.MapOutputBuffer.flush() method, we only released the kvbuffer before the merge phase (i.e., mergeParts()). Though kvindices and kvoffsets are small in many cases, we should release them theoretically before mergeParts().

try

{ spillThread.interrupt(); spillThread.join(); }

catch (InterruptedException e)

{ throw (IOException)new IOException("Spill failed" ).initCause(e); }

// release sort buffer before the merge
kvbuffer = null;
mergeParts();
Path outputPath = mapOutputFile.getOutputFile();
fileOutputByteCounter.increment(rfs.getFileStatus(outputPath).getLen());
}