Details
Major Description
Context:
KMS depends on hadoop-common for DT management. Recently we were analysing one performance issue and following is out findings:
- Around 96% (196 out of 200) KMS container threads were in BLOCKED state at following:
- AbstractDelegationTokenSecretManager.verifyToken()
- AbstractDelegationTokenSecretManager.createPassword()
- And then process crashed.
http-nio-9292-exec-200PRIORITY : 5THREAD ID : 0X00007F075C157800NATIVE ID : 0X2C87FNATIVE ID (DECIMAL) : 182399STATE : BLOCKED
stackTrace:
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.verifyToken(AbstractDelegationTokenSecretManager.java:474)
- waiting to lock <0x00000005f2f545e8> (a org.apache.hadoop.security.token.delegation.web.DelegationTokenManager$ZKSecretManager)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenManager.verifyToken(DelegationTokenManager.java:213)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:396)
at
All the 199 out of 200 were blocked at above point.
And the lock they are waiting for is acquired by a thread that was trying to createPassword and publishing the same on ZK.
stackTrace: java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1598) - locked <0x0000000749263ec0> (a org.apache.zookeeper.ClientCnxn$Packet) at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1570) at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:2235) at org.apache.curator.framework.imps.SetDataBuilderImpl$7.call(SetDataBuilderImpl.java:398) at org.apache.curator.framework.imps.SetDataBuilderImpl$7.call(SetDataBuilderImpl.java:385) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:93) at org.apache.curator.framework.imps.SetDataBuilderImpl.pathInForeground(SetDataBuilderImpl.java:382) at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:358) at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:36) at org.apache.curator.framework.recipes.shared.SharedValue.trySetValue(SharedValue.java:201) at org.apache.curator.framework.recipes.shared.SharedCount.trySetCount(SharedCount.java:116) at org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.incrSharedCount(ZKDelegationTokenSecretManager.java:586) at org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.incrementDelegationTokenSeqNum(ZKDelegationTokenSecretManager.java:601) at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.createPassword(AbstractDelegationTokenSecretManager.java:402) - locked <0x00000005f2f545e8> (a org.apache.hadoop.security.token.delegation.web.DelegationTokenManager$ZKSecretManager) at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.createPassword(AbstractDelegationTokenSecretManager.java:48) at org.apache.hadoop.security.token.Token.<init>(Token.java:67) at org.apache.hadoop.security.token.delegation.web.DelegationTokenManager.createToken(DelegationTokenManager.java:183)
We can say that this thread is slow and has blocked remaining all. But following is my observation:
- verifyToken() and createPaswword() has been synchronized because one is reading the tokenMap and another is updating the map. If it's only to protect the map, then can't we simply use ConcurrentHashMap and remove the "synchronized" keyword. Because due to this, all reader threads ( to verifyToken()) are also blocking each other.
- IN HA env, It is recommended to use ZK to store DTs. We know that CuratorFramework is thread safe. ZKDelegationTokenSecretManager.incrementDelegationTokenSeqNum() only requires to be protected from concurrent execution and it should be protected using some other locks instead of "this".
- With these changes, verifyToken() and createPaswword() will not block each other. It will be blocked only at the time of updating the map.
- Similarly other methods can also be considered but these two are critical.
I made these changes on my local and got the significant performance improvement.
I request community to provide their input and if we agree, I can provide the patch. Please let me know if any other details are required.
Thanks.
