This test is failing due to a deadlock.
When RM transitions to active, we store RM delegation token master key in state store. For this we put the state store event in AsyncDispatcher.
After event is picked up from AsyncDispatcher, we call RMStateStore#handleStoreEvent where we acquire a write lock. Then from StoreRMDTMasterKeyTransition, we will call MemoryRMStateStore#storeRMDTMasterKeyState which is a synchronized method.
Now in TestRMHA, we override updateApplicationState in MemoryRMStateStore which is also synchronized. By overriding this method, we are bypassing RMStateStore i.e. when in test we call rm.getRMContext().getStateStore().updateApplicationState(null), we do not try to acquire write lock in RMStateStore. When updateApplicationState calls notifyStoreOperationFailed, we will call RMStateStore#isFencedState which leads to acquiring of read lock or call RMStateStore#updateFencedState which will lead to acquiring of write lock.
Now due to race, if MemoryRMStateStore#updateApplicationState is called before MemoryRMStateStore#storeRMDTMasterKeyState is called but after RMStateStore#storeRMDTMasterKey is called, there can be a deadlock.
This is because the thread calling notifyStoreOperationFailed would be blocked while trying to acquire read or write lock in RMStateStore because a write lock is held by thread storing RM DT master key. Whereas the thread calling MemoryRMStateStore#storeRMDTMasterKeyState will be blocked because access to MemoryRMStateStore#updateApplicationState is synchronized and that thread is blocked on the read/write lock.
To solve this we should override updateApplicationStateInternal in MemoryRMStateStore and RMStateStore#updateApplicationState should be invoked so that normal flow of processing state store events is followed. This will get rid of deadlock.
This deadlock can be easily simulated by putting a sleep in StoreRMDTMasterKeyTransition#transition.