[MAPREDUCE-5186] mapreduce.job.max.split.locations causes some splits created by CombineFileInputFormat to fail

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.0.4-alpha, 2.2.0
• Fix Version/s: 2.3.0
• Component/s: job submission
• Target Version/s: 2.3.0
• Hadoop Flags: Reviewed
• Assignee: Robert Parker
• Reporter: Sangjin Lee
• Created: 27/Apr/13 00:28
• Updated: 03/Sep/14 23:35
• Resolved: 11/Nov/13 19:30

Description

CombineFileInputFormat can easily create splits that can come from many different locations (during the last pass of creating "global" splits). However, we observe that this often runs afoul of the mapreduce.job.max.split.locations check that's done by JobSplitWriter.

The default value for mapreduce.job.max.split.locations is 10, and with any decent size cluster, CombineFileInputFormat creates splits that are well above this limit.

Issue Links

• is related to: MAPREDUCE-1943 Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes
• is related to: MAPREDUCE-4146 Support limits on task status string length and number of block locations in branch-2