[HDDS-9263] Intermittent error in TestOzoneManagerHAMetadataOnly#testAllVolumeOperations

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: Ozone Manager
• Assignee: Arafat Khan
• Reporter: Arafat Khan
• Created: 11/Sep/23 07:04
• Updated: 11/Oct/23 07:19
• Resolved: 20/Sep/23 12:26

Description

After having multiple runs we encountered the following error in TestOzoneManagerHAMetadataOnly#testAllVolumeOperations
Test Runs :- https://github.com/ArafatKhan2198/ozone/actions/runs/6098706323

[INFO] Running org.apache.hadoop.ozone.om.TestOzoneManagerHAMetadataOnly 

4504Error:  Tests run: 11, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 152.877 s <<< FAILURE! - in org.apache.hadoop.ozone.om.TestOzoneManagerHAMetadataOnly 
4505Error:  org.apache.hadoop.ozone.om.TestOzoneManagerHAMetadataOnly.testAllVolumeOperations Time elapsed: 5.659 s <<< ERROR! 
4506org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ozone.om.exceptions.OMNotLeaderException): OM:omNode-1 is not the leader. Suggested leader is OM:omNode-2[localhost/127.0.0.1]. 
4507 at org.apache.hadoop.ozone.om.OzoneManager.checkLeaderStatus(OzoneManager.java:4014) 
4508 at org.apache.hadoop.ozone.om.ratis.utils.OzoneManagerRatisUtils.checkLeaderStatus(OzoneManagerRatisUtils.java:473) 
4509 at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:185) 
4510 at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87) 
4511 at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:148) 
4512 at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java) 
4513 at org.apache.hadoop.ipc.ProtobufRpcEngine$Server.processCall(ProtobufRpcEngine.java:484) 
4514 at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:595) 
4515 at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:573) 
4516 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1227) 
4517 at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1094) 
4518 at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1017) 
4519 at java.security.AccessController.doPrivileged(Native Method) 
4520 at javax.security.auth.Subject.doAs(Subject.java:422) 
4521 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) 
4522 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3048) 
4523
4524 at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1567) 
4525 at org.apache.hadoop.ipc.Client.call(Client.java:1513) 
4526 at org.apache.hadoop.ipc.Client.call(Client.java:1410) 
4527 at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:250) 
4528 at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:132) 
4529 at com.sun.proxy.$Proxy56.submitRequest(Unknown Source) 
4530 at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source) 
4531 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
4532 at java.lang.reflect.Method.invoke(Method.java:498) 
4533 at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:433) 
4534 at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:166) 
4535 at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:158) 
4536 at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:96) 
4537 at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:362) 
4538 at com.sun.proxy.$Proxy56.submitRequest(Unknown Source) 
4539 at org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransport.submitRequest(Hadoop3OmTransport.java:80) 
4540 at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:322) 
4541 at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.createVolume(OzoneManagerProtocolClientSideTranslatorPB.java:344) 
4542 at org.apache.hadoop.ozone.client.rpc.RpcClient.createVolume(RpcClient.java:465) 
4543 at org.apache.hadoop.ozone.client.ObjectStore.createVolume(ObjectStore.java:128) 
4544 at org.apache.hadoop.ozone.om.TestOzoneManagerHAMetadataOnly.createAndCheckVolume(TestOzoneManagerHAMetadataOnly.java:87) 
4545 at org.apache.hadoop.ozone.om.TestOzoneManagerHAMetadataOnly.testAllVolumeOperations(TestOzoneManagerHAMetadataOnly.java:103)
 

 

https://github.com/apache/ozone/pull/5329

Issue Links

• is cloned by: HDDS-9429 Intermittent error in TestOzoneManagerHAMetadataOnly#testAllVolumeOperations
• supercedes: HDDS-6644 Intermittent failure in TestOzoneManagerHAMetadataOnly#testReadRequest