[HADOOP-874] merge code is really slow

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.10.0
• Component/s: io
• Assignee: Devaraj Das
• Reporter: Owen O'Malley
• Created: 10/Jan/07 16:32
• Updated: 23/Apr/09 19:24
• Resolved: 08/Apr/09 17:07

Description

I had a case where the map output buffer size (io.sort.mb) was set too low and caused a spill and merge. Fixing the configuration caused it to not spill until it was finished. With the spill it took 9.5 minutes per a map. Without the spill it took 45 seconds. Therefore, I assume it was taking ~9 minutes to do the 2 file merge. That is really slow. The input files to the merge were two 25 mb sequence files (default codec (java), block compressed)