[HDDS-5248] SCM HA: Continuous PipelineNotFoundException seen in SCM log

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: SCM, SCM HA
• Labels: pull-request-available
• Target Version/s: 1.2.0
• Assignee: Lokesh Jain
• Reporter: Nilotpal Nandi
• Created: 19/May/21 10:10
• Updated: 24/May/21 06:04
• Resolved: 24/May/21 06:04

Description

After running decommissioning tests and aborting few decommissioning command, seeing PipelineNotFoundException INFO logs continuously in scm log.
Although these are INFO logs, raising the issue as the exceptions are thrown in loop.

2021-05-16 20:06:53,421 INFO org.apache.hadoop.ipc.Server: IPC Server handler 53 on 9860, call Call#102 Retry#23 org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocol.submitRequest from 172.27.176.196:41982
org.apache.hadoop.hdds.scm.pipeline.PipelineNotFoundException: PipelineID=b1431673-adf0-402f-968f-e9241ea37609 not found
	at org.apache.hadoop.hdds.scm.pipeline.PipelineStateMap.getPipeline(PipelineStateMap.java:121)
	at org.apache.hadoop.hdds.scm.pipeline.PipelineStateManagerV2Impl.getPipeline(PipelineStateManagerV2Impl.java:125)
	at jdk.internal.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invokeLocal(SCMHAInvocationHandler.java:83)
	at org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invoke(SCMHAInvocationHandler.java:68)
	at com.sun.proxy.$Proxy14.getPipeline(Unknown Source)
	at org.apache.hadoop.hdds.scm.pipeline.PipelineManagerV2Impl.getPipeline(PipelineManagerV2Impl.java:198)
	at org.apache.hadoop.hdds.scm.server.SCMClientProtocolServer.getPipeline(SCMClientProtocolServer.java:529)
	at org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.getPipeline(StorageContainerLocationProtocolServerSideTranslatorPB.java:497)
	at org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.processRequest(StorageContainerLocationProtocolServerSideTranslatorPB.java:302)
	at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
	at org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.submitRequest(StorageContainerLocationProtocolServerSideTranslatorPB.java:153)
	at org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos$StorageContainerLocationProtocolService$2.callBlockingMethod(StorageContainerLocationProtocolProtos.java:46400)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:533)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:986)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:914)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2887)
2021-05-16 20:06:55,426 INFO org.apache.hadoop.ipc.Server: IPC Server handler 84 on 9860, call Call#102 Retry#24 org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocol.submitRequest from 172.27.176.196:41982
org.apache.hadoop.hdds.scm.pipeline.PipelineNotFoundException: PipelineID=b1431673-adf0-402f-968f-e9241ea37609 not found
	at org.apache.hadoop.hdds.scm.pipeline.PipelineStateMap.getPipeline(PipelineStateMap.java:121)
	at org.apache.hadoop.hdds.scm.pipeline.PipelineStateManagerV2Impl.getPipeline(PipelineStateManagerV2Impl.java:125)
	at jdk.internal.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invokeLocal(SCMHAInvocationHandler.java:83)
	at org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invoke(SCMHAInvocationHandler.java:68)
	at com.sun.proxy.$Proxy14.getPipeline(Unknown Source)
	at org.apache.hadoop.hdds.scm.pipeline.PipelineManagerV2Impl.getPipeline(PipelineManagerV2Impl.java:198)
	at org.apache.hadoop.hdds.scm.server.SCMClientProtocolServer.getPipeline(SCMClientProtocolServer.java:529)
	at org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.getPipeline(StorageContainerLocationProtocolServerSideTranslatorPB.java:497)
	at org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.processRequest(StorageContainerLocationProtocolServerSideTranslatorPB.java:302)
	at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
	at org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.submitRequest(StorageContainerLocationProtocolServerSideTranslatorPB.java:153)
	at org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos$StorageContainerLocationProtocolService$2.callBlockingMethod(StorageContainerLocationProtocolProtos.java:46400)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:533)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:986)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:914)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2887)

Issue Links

• links to: GitHub Pull Request #2267