Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-2823 SCM HA Support
  3. HDDS-5248

SCM HA: Continuous PipelineNotFoundException seen in SCM log

    XMLWordPrintableJSON

    Details

    • Target Version/s:

      Description

      After running decommissioning tests and aborting few decommissioning command, seeing PipelineNotFoundException INFO logs continuously in scm log.
      Although these are INFO logs, raising the issue as the exceptions are thrown in loop.

      2021-05-16 20:06:53,421 INFO org.apache.hadoop.ipc.Server: IPC Server handler 53 on 9860, call Call#102 Retry#23 org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocol.submitRequest from 172.27.176.196:41982
      org.apache.hadoop.hdds.scm.pipeline.PipelineNotFoundException: PipelineID=b1431673-adf0-402f-968f-e9241ea37609 not found
      	at org.apache.hadoop.hdds.scm.pipeline.PipelineStateMap.getPipeline(PipelineStateMap.java:121)
      	at org.apache.hadoop.hdds.scm.pipeline.PipelineStateManagerV2Impl.getPipeline(PipelineStateManagerV2Impl.java:125)
      	at jdk.internal.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
      	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
      	at org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invokeLocal(SCMHAInvocationHandler.java:83)
      	at org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invoke(SCMHAInvocationHandler.java:68)
      	at com.sun.proxy.$Proxy14.getPipeline(Unknown Source)
      	at org.apache.hadoop.hdds.scm.pipeline.PipelineManagerV2Impl.getPipeline(PipelineManagerV2Impl.java:198)
      	at org.apache.hadoop.hdds.scm.server.SCMClientProtocolServer.getPipeline(SCMClientProtocolServer.java:529)
      	at org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.getPipeline(StorageContainerLocationProtocolServerSideTranslatorPB.java:497)
      	at org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.processRequest(StorageContainerLocationProtocolServerSideTranslatorPB.java:302)
      	at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
      	at org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.submitRequest(StorageContainerLocationProtocolServerSideTranslatorPB.java:153)
      	at org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos$StorageContainerLocationProtocolService$2.callBlockingMethod(StorageContainerLocationProtocolProtos.java:46400)
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:533)
      	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
      	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:986)
      	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:914)
      	at java.base/java.security.AccessController.doPrivileged(Native Method)
      	at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
      	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2887)
      2021-05-16 20:06:55,426 INFO org.apache.hadoop.ipc.Server: IPC Server handler 84 on 9860, call Call#102 Retry#24 org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocol.submitRequest from 172.27.176.196:41982
      org.apache.hadoop.hdds.scm.pipeline.PipelineNotFoundException: PipelineID=b1431673-adf0-402f-968f-e9241ea37609 not found
      	at org.apache.hadoop.hdds.scm.pipeline.PipelineStateMap.getPipeline(PipelineStateMap.java:121)
      	at org.apache.hadoop.hdds.scm.pipeline.PipelineStateManagerV2Impl.getPipeline(PipelineStateManagerV2Impl.java:125)
      	at jdk.internal.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
      	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
      	at org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invokeLocal(SCMHAInvocationHandler.java:83)
      	at org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invoke(SCMHAInvocationHandler.java:68)
      	at com.sun.proxy.$Proxy14.getPipeline(Unknown Source)
      	at org.apache.hadoop.hdds.scm.pipeline.PipelineManagerV2Impl.getPipeline(PipelineManagerV2Impl.java:198)
      	at org.apache.hadoop.hdds.scm.server.SCMClientProtocolServer.getPipeline(SCMClientProtocolServer.java:529)
      	at org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.getPipeline(StorageContainerLocationProtocolServerSideTranslatorPB.java:497)
      	at org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.processRequest(StorageContainerLocationProtocolServerSideTranslatorPB.java:302)
      	at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
      	at org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.submitRequest(StorageContainerLocationProtocolServerSideTranslatorPB.java:153)
      	at org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos$StorageContainerLocationProtocolService$2.callBlockingMethod(StorageContainerLocationProtocolProtos.java:46400)
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:533)
      	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
      	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:986)
      	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:914)
      	at java.base/java.security.AccessController.doPrivileged(Native Method)
      	at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
      	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2887)
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ljain Lokesh Jain
                Reporter:
                nilotpalnandi Nilotpal Nandi
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: