[MAPREDUCE-5744] Job hangs because RMContainerAllocator$AssignedRequests.preemptReduce() violates the comparator contract

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.0.5-alpha
• Fix Version/s: 0.23.11, 2.3.0
• Target Version/s: 2.3.0
• Hadoop Flags: Reviewed
• Assignee: Gera Shegalov
• Reporter: Sangjin Lee
• Created: 06/Feb/14 19:05
• Updated: 28/Feb/14 21:35
• Resolved: 06/Feb/14 23:34

Description

We ran into a situation where tasks are not getting assigned because RMContainerAllocator$AssignedRequests.preemptReduce() fails repeatedly with the following exception:

2014-02-06 16:43:45,183 ERROR [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: ERROR IN CONTACTING RM.
java.lang.IllegalArgumentException: Comparison method violates its general contract!
     at java.util.TimSort.mergeLo(TimSort.java:747)
     at java.util.TimSort.mergeAt(TimSort.java:483)
     at java.util.TimSort.mergeCollapse(TimSort.java:408)
     at java.util.TimSort.sort(TimSort.java:214)
     at java.util.TimSort.sort(TimSort.java:173)
     at java.util.Arrays.sort(Arrays.java:659)
     at java.util.Collections.sort(Collections.java:217)
     at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator$AssignedRequests.preemptReduce(RMContainerAllocator.java:1106)
     at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.preemptReducesIfNeeded(RMContainerAllocator.java:416)
     at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:230)
     at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$1.run(RMCommunicator.java:252)
     at java.lang.Thread.run(Thread.java:744)

It is because the comparator that's defined in this method does not abide by the contract, specifically if p == 0.

Comparator.compare(): http://docs.oracle.com/javase/7/docs/api/java/util/Comparator.html#compare(T, T)