[YARN-10765] Fair scheduler continuous scheduling thread crashes while sorting nodes

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.7.0, 2.7.1
• Fix Version/s: 3.0.0
• Component/s: scheduler
• Target Version/s: 2.7.0
• Flags: Patch
• Assignee: Unassigned
• Reporter: jianjin qin
• Created: 10/May/21 05:22
• Updated: 10/May/21 06:48

Description

NodeAvailableResourceComparator use variable nodes which contains to compare, but nodeIdList is not same with nodes.keySet() during sort. when node removed before sort, such as n2,n3 are removed, compare(n2, n3)=1 and compare(n3, n2)=1, this violates the reflexivity of comparator, we may get the exception: java.lang.IllegalArgumentException: Comparison method violates its general contract!

2021-01-19 10:22:55,102 ERROR org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread Thread[FairSchedulerContinuousScheduling,5,main] threw an Exception.
java.lang.IllegalArgumentException: Comparison method violates its general contract!
        at java.util.TimSort.mergeLo(TimSort.java:747)
        at java.util.TimSort.mergeAt(TimSort.java:483)
        at java.util.TimSort.mergeCollapse(TimSort.java:410)
        at java.util.TimSort.sort(TimSort.java:214)
        at java.util.TimSort.sort(TimSort.java:173)
        at java.util.Arrays.sort(Arrays.java:659)
        at java.util.Collections.sort(Collections.java:217)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.continuousSchedulingAttempt(FairScheduler.java:1069)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$ContinuousSchedulingThread.run(FairScheduler.java:322)