[MAPREDUCE-7132] JobSplitWriter prints unnecessary warnings if EC(RS10,4) is used

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.1
• Fix Version/s: 3.2.0, 3.0.4, 3.1.2
• Component/s: client, mrv2
• Hadoop Flags: Reviewed
• Assignee: Peter Bacsko
• Reporter: Peter Bacsko
• Created: 29/Aug/18 11:26
• Updated: 16/Oct/18 17:49
• Resolved: 16/Oct/18 17:25

Description

Currently, JobSplitWriter compares the number of hosts for a certain block against a static value that comes from mapreduce.job.max.split.locations. The default value of this property is 10.

However, an EC schema like RS-10-4 requires at least 14 hosts. In this case, 14 block locations will be returned and JobSplitWriter prints a warning, which can confuse users.

A possible solution could check whether EC is enabled for a block and increase this value dynamically if needed. A simpler one is to simply increase the default value to a sensible number like 15.