[HDDS-10886] Intermittent timeout in TestOzoneRpcClientWithRatis#testParallelDeleteBucketAndCreateKey

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: test
• Assignee: Unassigned
• Reporter: Attila Doroszlai
• Created: 21/May/24 04:49
• Updated: 10/Jul/24 09:27

Description

testParallelDeleteBucketAndCreateKey() TIMED OUT. PRINTING THREAD DUMP

"Thread-3221"  prio=5 tid=4062 in Object.wait()
java.lang.Thread.State: WAITING (on object monitor)
        at java.base@17.0.10/java.lang.Object.wait(Native Method)
        ...
        at app//org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1555)
        ...
        at app//org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransport.submitRequest(Hadoop3OmTransport.java:80)
        at app//org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:336)
        at app//org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleSubmitRequestAndSCMSafeModeRetry(OzoneManagerProtocolClientSideTranslatorPB.java:2256)
        at app//org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.openKey(OzoneManagerProtocolClientSideTranslatorPB.java:735)
        at app//org.apache.hadoop.ozone.client.rpc.RpcClient.createKey(RpcClient.java:1423)
        at app//org.apache.hadoop.ozone.client.rpc.RpcClient.createKey(RpcClient.java:1380)
        at app//org.apache.hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis.lambda$testParallelDeleteBucketAndCreateKey$1(TestOzoneRpcClientWithRatis.java:315)

"Thread-3220"  prio=5 tid=4061 in Object.wait()
java.lang.Thread.State: WAITING (on object monitor)
        at java.base@17.0.10/java.lang.Object.wait(Native Method)
        ...
        at app//org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1555)
        ...
        at app/jdk.proxy2/jdk.proxy2.$Proxy69.submitRequest(Unknown Source)
        at app//org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransport.submitRequest(Hadoop3OmTransport.java:80)
        at app//org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:336)
        at app//org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.deleteBucket(OzoneManagerProtocolClientSideTranslatorPB.java:982)
        at app//org.apache.hadoop.ozone.client.rpc.RpcClient.deleteBucket(RpcClient.java:1303)
        at app//org.apache.hadoop.ozone.client.OzoneVolume.deleteBucket(OzoneVolume.java:396)
        at app//org.apache.hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis.lambda$testParallelDeleteBucketAndCreateKey$0(TestOzoneRpcClientWithRatis.java:307)


"IPC Server handler 16 on default port 15004" daemon prio=5 tid=432 in Object.wait()
java.lang.Thread.State: WAITING (on object monitor)
        ...
        at java.base@17.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2072)
        at app//org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisServer.submitRequestToRatisImpl(OzoneManagerRatisServer.java:320)
        ...
        at app//org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisServer.submitRequest(OzoneManagerRatisServer.java:262)
        at app//org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequestToRatis(OzoneManagerProtocolServerSideTranslatorPB.java:252)

"om1-OMStateMachineApplyTransactionThread - 0" daemon prio=5 tid=1422 in Object.wait()
java.lang.Thread.State: WAITING (on object monitor)
        ...
        at java.base@17.0.10/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:230)
        at app//org.apache.hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis$OMRequestHandlerPauseInjector.pause(TestOzoneRpcClientWithRatis.java:372)
        at app//org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.injectPause(OzoneManagerRequestHandler.java:418)
        at app//org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequestImpl(OzoneManagerRequestHandler.java:392)

• https://github.com/adoroszlai/ozone-build-results/blob/master/2024/05/12/31254/it-client/hadoop-ozone/integration-test/org.apache.hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis-output.txt
• https://github.com/adoroszlai/ozone-build-results/blob/master/2024/05/19/31366/it-client/hadoop-ozone/integration-test/org.apache.hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis-output.txt

Issue Links

• is related to: HDDS-10143 Intermittent failure in testParallelDeleteBucketAndCreateKey