[HDDS-5381] SCM terminated with exit status 1: null

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: SCM HA
• Labels: pull-request-available
• Assignee: Shashikant Banerjee
• Reporter: Nilotpal Nandi
• Created: 24/Jun/21 11:04
• Updated: 14/Jul/21 14:48
• Resolved: 14/Jul/21 14:48

Description

steps taken :
Running ozone fault injection tests which involve activating/deactivating pipelines.
In one of the tests, while activating pipeline, SCM terminated.SCM log:

2021-06-23 23:57:03,166 WARN org.apache.hadoop.hdds.scm.pipeline.PipelineStateManager: Pipeline PipelineID=661b906a-fd6b-4c14-b6b5-8c0a12c67ff7 is not found in the pipeline Map. Pipeline may have been deleted already.
2021-06-23 23:57:03,168 ERROR org.apache.ratis.statemachine.StateMachine: Terminating with exit status 1: null
java.lang.IllegalMonitorStateException
	at java.base/java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryRelease(ReentrantReadWriteLock.java:372)
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1302)
	at java.base/java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.unlock(ReentrantReadWriteLock.java:1147)
	at org.apache.hadoop.hdds.scm.pipeline.PipelineStateManagerV2Impl.updatePipelineState(PipelineStateManagerV2Impl.java:270)
	at jdk.internal.reflect.GeneratedMethodAccessor101.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.hadoop.hdds.scm.ha.SCMStateMachine.process(SCMStateMachine.java:168)
	at org.apache.hadoop.hdds.scm.ha.SCMStateMachine.applyTransaction(SCMStateMachine.java:139)
	at org.apache.ratis.server.impl.RaftServerImpl.applyLogToStateMachine(RaftServerImpl.java:1690)
	at org.apache.ratis.server.impl.StateMachineUpdater.applyLog(StateMachineUpdater.java:234)
	at org.apache.ratis.server.impl.StateMachineUpdater.run(StateMachineUpdater.java:179)
	at java.base/java.lang.Thread.run(Thread.java:834)
2021-06-23 23:57:03,174 INFO org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter: SHUTDOWN_MSG: 

Issue Links

• links to: GitHub Pull Request #2362