Optimize MapTask.MapOutputBuffer.spill() by not deserialize/serialize keys/values but use appendRaw

• Key: HADOOP-1609
• Type: Improvement
• Status: Closed
• Resolution: Duplicate
• Priority: Major
• Assignee: Unassigned
• Reporter: Espen Amble Kolstad
• Affects Version/s: 0.14.0

Description

In MapTask.MapOutputBuffer.spill() every key and value is read from buffer and then written to file with append(key, value):

DataInputBuffer keyIn = new DataInputBuffer();
      DataInputBuffer valIn = new DataInputBuffer();
      DataOutputBuffer valOut = new DataOutputBuffer();
      while (resultIter.next()) {
        keyIn.reset(resultIter.getKey().getData(), 
                    resultIter.getKey().getLength());
        key.readFields(keyIn);
        valOut.reset();
        (resultIter.getValue()).writeUncompressedBytes(valOut);
        valIn.reset(valOut.getData(), valOut.getLength());
        value.readFields(valIn);
        writer.append(key, value);
        reporter.progress();
      }

When you have complex objects, like nutch's ParseData or Inlinks, this takes time and creates lots of garbage.

I've created a patch, it seems to be working, only tested on 0.13.0.
It's a bit clumsy, since ValueBytes is cast to Un-/CompressedBytes in SequenceFile.Writer.

Thoughts?