[MAPREDUCE-5966] MR1 FairScheduler use of custom weight adjuster is not thread safe for comparisons

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.1
• Fix Version/s: 1.3.0
• Component/s: scheduler
• Target Version/s: 1.3.0
• Hadoop Flags: Reviewed
• Assignee: Anubhav Dhoot
• Reporter: Anubhav Dhoot
• Created: 11/Jul/14 00:22
• Updated: 26/Jul/14 19:58
• Resolved: 26/Jul/14 19:54

Description

When comparing JobSchedulables one of the factors is the weight. If someone uses a custom weight adjuster, that may be called multiple times during a sort causing different values to return. That causes a failure in sorting because the weight may change during the sort.

This reproes as

java.io.IOException: java.lang.IllegalArgumentException: Comparison method violates its general contract!
at java.util.TimSort.mergeHi(TimSort.java:868)
at java.util.TimSort.mergeAt(TimSort.java:485)
at java.util.TimSort.mergeCollapse(TimSort.java:410)
at java.util.TimSort.sort(TimSort.java:214)
at java.util.TimSort.sort(TimSort.java:173)
at java.util.Arrays.sort(Arrays.java:659)
at java.util.Collections.sort(Collections.java:217)
at org.apache.hadoop.mapred.PoolSchedulable.assignTask(PoolSchedulable.java:163)
at org.apache.hadoop.mapred.FairScheduler.assignTasks(FairScheduler.java:499)
at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2961)