Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-4850

Intermittent failure in ozonesecure due to unable to allocate block

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.1.0
    • Component/s: None
    • Target Version/s:

      Description

      scm_1       | 2021-02-19 02:23:28,815 [IPC Server handler 28 on default port 9863] WARN block.BlockManagerImpl: Pipeline creation failed for type:RATIS factor:THREE. Datanodes may be used up.
      scm_1       | org.apache.hadoop.hdds.scm.exceptions.SCMException: Pipeline creation failed because nodes are engaged in other pipelines and every node can only be engaged in max 1 pipelines. Required 3. Found 0
      scm_1       | 	at org.apache.hadoop.hdds.scm.pipeline.PipelinePlacementPolicy.filterViableNodes(PipelinePlacementPolicy.java:172)
      scm_1       | 	at org.apache.hadoop.hdds.scm.pipeline.PipelinePlacementPolicy.chooseDatanodes(PipelinePlacementPolicy.java:224)
      scm_1       | 	at org.apache.hadoop.hdds.scm.pipeline.RatisPipelineProvider.create(RatisPipelineProvider.java:134)
      scm_1       | 	at org.apache.hadoop.hdds.scm.pipeline.PipelineFactory.create(PipelineFactory.java:63)
      scm_1       | 	at org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.createPipeline(SCMPipelineManager.java:271)
      scm_1       | 	at org.apache.hadoop.hdds.scm.block.BlockManagerImpl.allocateBlock(BlockManagerImpl.java:201)
      scm_1       | 	at org.apache.hadoop.hdds.scm.server.SCMBlockProtocolServer.allocateBlock(SCMBlockProtocolServer.java:192)
      scm_1       | 	at org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.allocateScmBlock(ScmBlockLocationProtocolServerSideTranslatorPB.java:162)
      scm_1       | 	at org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.processMessage(ScmBlockLocationProtocolServerSideTranslatorPB.java:118)
      scm_1       | 	at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
      scm_1       | 	at org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.send(ScmBlockLocationProtocolServerSideTranslatorPB.java:100)
      scm_1       | 	at org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos$ScmBlockLocationProtocolService$2.callBlockingMethod(ScmBlockLocationProtocolProtos.java:13265)
      scm_1       | 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:528)
      scm_1       | 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1086)
      scm_1       | 	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1029)
      scm_1       | 	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:957)
      scm_1       | 	at java.base/java.security.AccessController.doPrivileged(Native Method)
      scm_1       | 	at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
      scm_1       | 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
      scm_1       | 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2957)
      scm_1       | 2021-02-19 02:23:28,815 [IPC Server handler 28 on default port 9863] ERROR block.BlockManagerImpl: Unable to allocate a block for the size: 268435456, type: RATIS, factor: THREE
      

      https://github.com/elek/ozone-build-results/tree/master/2021/02/19/6000/acceptance-secure

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                adoroszlai Attila Doroszlai
                Reporter:
                adoroszlai Attila Doroszlai
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: