[YARN-3091] [Umbrella] Improve and fix locks of RM scheduler

• Type: Task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: capacityscheduler, fairscheduler, resourcemanager, scheduler
• Assignee: Unassigned
• Reporter: Wangda Tan
• Created: 22/Jan/15 23:02
• Updated: 02/Jul/18 10:13

Description

In existing YARN RM scheduler, there're some issues of using locks. For example:

• Many unnecessary synchronized locks, we have seen several cases recently that too frequent access of scheduler makes scheduler hang. Which could be addressed by using read/write lock. Components include scheduler, CS queues, apps
• Some fields not properly locked (Like clusterResource)

We can address them together in this ticket.

(More details see comments below)

Issue Links

• is duplicated by: YARN-5964 Lower the granularity of locks in FairScheduler
• is related to: YARN-3139 Improve locks in AbstractYarnScheduler/CapacityScheduler/FairScheduler
• is related to: HADOOP-9213 Create a Jenkins job to run jcarder

Sub-Tasks

• CapacityScheduler.checkAccess unnecessarily grabs the scheduler lock: Open
• getTransferredContainers can be a bottleneck during AM registration: Closed
• Improve locks in AbstractYarnScheduler/CapacityScheduler/FairScheduler: Resolved
• Improve locks in AbstractCSQueue/LeafQueue/ParentQueue: Resolved
• Improve locks in SchedulerApplicationAttempt/FSAppAttempt/FiCaSchedulerApp: Resolved
• Improve locks in AppSchedulingInfo: Resolved
• FairScheduler: Use lock for queuemanager instead of synchronized on FairScheduler: Open
• CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue: Closed
• Deadlock due to synchronised get Methods in AbstractCSQueue: Resolved
• Improve locks of OrderingPolicy: Open
• Fix synchronization issues of AbstractYarnScheduler#nodeUpdate and its implementations: Resolved
• Introduce AtomicResource in ResourceUsage to avoid read-write lock.: Open