[MAPREDUCE-3710] last split generated by FileInputFormat.getSplits may not have the best locality

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.23.0, 1.0.0
• Fix Version/s: 0.23.1
• Component/s: mrv1, mrv2
• Hadoop Flags: Reviewed
• Release Note: Improved FileInputFormat to return better locality for the last split.
• Assignee: Siddharth Seth
• Reporter: Siddharth Seth
• Created: 23/Jan/12 19:13
• Updated: 05/Mar/12 02:49
• Resolved: 24/Jan/12 21:32

Description

The last split generated by FileInputFormat.getSplits considers blkLocations.length-1 to be the hosts for the split.
The last split may be larger than the rest (SPLIT_SLOP=1.1 by default) - in which case locality is picked up from a smaller block.
e.g. 1027MB file with a 128MB split size. The last split ends up being 131MB. The hosts for locality end up being the nodes containing the 3MB block instead of the 128MB block.

Issue Links

• is related to: MAPREDUCE-3524 Scan benchmark is more than 1.5x slower in 0.23