[HDDS-4850] Intermittent failure in ozonesecure due to unable to allocate block

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.1.0
• Labels: pull-request-available
• Target Version/s: 1.1.0
• Assignee: Attila Doroszlai
• Reporter: Attila Doroszlai
• Created: 19/Feb/21 15:10
• Updated: 22/Feb/21 19:16
• Resolved: 22/Feb/21 14:22

Description

scm_1       | 2021-02-19 02:23:28,815 [IPC Server handler 28 on default port 9863] WARN block.BlockManagerImpl: Pipeline creation failed for type:RATIS factor:THREE. Datanodes may be used up.
scm_1       | org.apache.hadoop.hdds.scm.exceptions.SCMException: Pipeline creation failed because nodes are engaged in other pipelines and every node can only be engaged in max 1 pipelines. Required 3. Found 0
scm_1       | 	at org.apache.hadoop.hdds.scm.pipeline.PipelinePlacementPolicy.filterViableNodes(PipelinePlacementPolicy.java:172)
scm_1       | 	at org.apache.hadoop.hdds.scm.pipeline.PipelinePlacementPolicy.chooseDatanodes(PipelinePlacementPolicy.java:224)
scm_1       | 	at org.apache.hadoop.hdds.scm.pipeline.RatisPipelineProvider.create(RatisPipelineProvider.java:134)
scm_1       | 	at org.apache.hadoop.hdds.scm.pipeline.PipelineFactory.create(PipelineFactory.java:63)
scm_1       | 	at org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.createPipeline(SCMPipelineManager.java:271)
scm_1       | 	at org.apache.hadoop.hdds.scm.block.BlockManagerImpl.allocateBlock(BlockManagerImpl.java:201)
scm_1       | 	at org.apache.hadoop.hdds.scm.server.SCMBlockProtocolServer.allocateBlock(SCMBlockProtocolServer.java:192)
scm_1       | 	at org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.allocateScmBlock(ScmBlockLocationProtocolServerSideTranslatorPB.java:162)
scm_1       | 	at org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.processMessage(ScmBlockLocationProtocolServerSideTranslatorPB.java:118)
scm_1       | 	at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
scm_1       | 	at org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.send(ScmBlockLocationProtocolServerSideTranslatorPB.java:100)
scm_1       | 	at org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos$ScmBlockLocationProtocolService$2.callBlockingMethod(ScmBlockLocationProtocolProtos.java:13265)
scm_1       | 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:528)
scm_1       | 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1086)
scm_1       | 	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1029)
scm_1       | 	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:957)
scm_1       | 	at java.base/java.security.AccessController.doPrivileged(Native Method)
scm_1       | 	at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
scm_1       | 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
scm_1       | 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2957)
scm_1       | 2021-02-19 02:23:28,815 [IPC Server handler 28 on default port 9863] ERROR block.BlockManagerImpl: Unable to allocate a block for the size: 268435456, type: RATIS, factor: THREE

https://github.com/elek/ozone-build-results/tree/master/2021/02/19/6000/acceptance-secure

Issue Links

• is duplicated by: HDDS-4852 acceptance (secure) fails frequently at `ozone freon ockg -n1 -t1 -p container` in container.robot
• links to: GitHub Pull Request #1948