[MAPREDUCE-2910] Allow empty MapOutputFile segments

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 0.20.2, 0.23.0
• Component/s: task, tasktracker
• Assignee: Unassigned
• Reporter: Binglin Chang
• Created: 30/Aug/11 03:47
• Updated: 10/Mar/15 01:29

Description

As the scale of cluster and job get larger, we see a lot of empty partitions in MapOutputFile due to large reduce numbers or partition skew. When map output compression is enabled, empty map output partitions gets larger & has additional compressor/decompressor initialization overhead.
This can be optimized by allowing empty MapOutputFile segments, where the rawLength & partLength of IndexRecord all equal to 0. Corresponding support need to be added to IFile reader, writer, and reduce shuffle copier.