[YARN-10347] Fix double locking in CapacityScheduler#reinitialize in branch-3.1

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.1.4
• Fix Version/s: 3.2.2, 3.1.4
• Component/s: capacity scheduler
• Target Version/s: 3.1.4
• Assignee: Masatake Iwasaki
• Reporter: Masatake Iwasaki
• Created: 08/Jul/20 14:20
• Updated: 14/Jul/20 05:44
• Resolved: 14/Jul/20 05:44

Description

Double locking blocks another threads in ResourceManager waiting for the lock.

I found the issue on testing hadoop-3.1.4-RC2 with RM-HA enabled deployment. ResourceManager blocks on submitApplication waiting for the lock when I run example MR applications.

"IPC Server handler 45 on default port 8032" #211 daemon prio=5 os_prio=0 tid=0x00007f0e45a40200 nid=0x418 waiting on condition [0x00007f0e14abe000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x0000000085d56510> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.checkAndGetApplicationPriority(CapacityScheduler.java:2521)
        at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:417)
        at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:342)
        at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:678)
        at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:277)
        at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:563)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943)

 

Issue Links

• duplicates: YARN-9341 Reentrant lock() before try
• is caused by: YARN-10022 Create RM Rest API to validate a CapacityScheduler Configuration
• links to: GitHub Pull Request #2129