map outputs should not be renamed between partitions

• Key: HADOOP-3443
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Critical
• Assignee: Owen O'Malley
• Reporter: Owen O'Malley
• Affects Version/s: 0.17.0
• Fix Version/s: 0.18.0

Description

If a map finishes with out having to spill its data buffer, the map outputs are sorted and written to disk. However, no care is taken to make sure that the same partition is used to write it out before it is renamed. On nodes with multiple disks assigned to the task trackers, this will likely cause an addition read/write cycle to disk that is very expensive.

Comments

[Owen O'Malley]

This patch uses the same directory for the final outputs in the case that there is a single spill. It also adds some logging so you can understand if (and why) the mapper's sort spilled.

[Owen O'Malley]

Should this be fixed in 0.17?

[Doug Cutting]

> Should this be fixed in 0.17?

Is it a regression?

[Chris Douglas]

+1 Looks good. Some of the logging could be at the debug level, particularly the logging in the synchronized section of BlockingBuffer.Buffer::write, but it shouldn't be a real issue.

Is it a regression?

No; it looks pretty ancient, but if there's a 0.17.1, including this would only make things better.

[Chris Douglas]

Port to 0.17, if we need it.

[Chris Douglas]

I just committed this. Thanks, Owen

[Hudson]

Integrated in Hadoop-trunk #511 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/511/)