[MAPREDUCE-26] The shuffle keeps the ReduceTask locked while doing a FileSystem.rename leading to task timeouts

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Assignee: Owen O'Malley
• Reporter: Owen O'Malley
• Created: 24/Aug/07 18:31
• Updated: 31/Dec/11 09:10
• Resolved: 31/Dec/11 09:10

Description

The shuffle in ReduceTask.ReduceCopier.MapOutputCopier.copyOutput locks the entire ReduceTask while doing a FileSystem.rename operation. Unfortunately the RawLocalFileSystem implements rename as a copy and delete, which can take a long time. As a result the reduce is being killed as not reporting progress for 10 minutes.