[MAPREDUCE-7085] while loop in InputSampler.writePartitionFile method is never entered

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 2.8.3
• Component/s: mrv2
• Assignee: Unassigned
• Reporter: Oliver Hummel
• Created: 22/Apr/18 15:20
• Updated: 22/Apr/18 15:23

Description

After getting the split points are out of order exception from TotalOrderPartitioner, I dug into the source of the org.apache.hadoop.mapreduce.lib.partition.InputSampler class and found that the while loop in line 335 is never entered.

The reason is that the variable last is always smaller than k and the loop condition says that last must be larger or equal than k.

I am not completely sure of the initial purpose of this loop, if it is what I assume, namely reducing the occurrences of identical split points, then I would change it like so:

  while (last != -1 && k > last && comparator.compare(samples[last], samples[k]) == 0)

{         --k;              }

However, this only slightly mitigates the problem, since a highly skewed distribution of keys still might lead to identical split points so that potentially further measures might be necessary?