[MAPREDUCE-5352] Optimize node local splits generated by CombineFileInputFormat

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.5-alpha
• Fix Version/s: 2.1.0-beta
• Hadoop Flags: Reviewed
• Assignee: Siddharth Seth
• Reporter: Siddharth Seth
• Created: 25/Jun/13 04:46
• Updated: 27/Aug/13 22:22
• Resolved: 01/Aug/13 17:48

Description

CombineFileInputFormat currently walks through all available nodes and generates multiple (maxSplitsPerNode) splits on a single node before attempting to generate splits on subsequent nodes. This ends up reducing the possibility of generating splits for subsequent nodes - since these blocks will no longer be available for subsequent nodes. Allowing splits to go 1 block above the max-split-size makes this worse.
Allocating a single split per node in one iteration, should help increase the distribution of splits across nodes - so the subsequent nodes will have more blocks to choose from.