Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-5626 Track and Address Flaky tests
  3. HDDS-9263

Intermittent error in TestOzoneManagerHAMetadataOnly#testAllVolumeOperations

Attach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.4.0
    • Ozone Manager
    • None

    Description

      After having multiple runs we encountered the following error in TestOzoneManagerHAMetadataOnly#testAllVolumeOperations
      Test Runs :- https://github.com/ArafatKhan2198/ozone/actions/runs/6098706323

      [INFO] Running org.apache.hadoop.ozone.om.TestOzoneManagerHAMetadataOnly 
      
      4504Error:  Tests run: 11, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 152.877 s <<< FAILURE! - in org.apache.hadoop.ozone.om.TestOzoneManagerHAMetadataOnly 
      4505Error:  org.apache.hadoop.ozone.om.TestOzoneManagerHAMetadataOnly.testAllVolumeOperations Time elapsed: 5.659 s <<< ERROR! 
      4506org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ozone.om.exceptions.OMNotLeaderException): OM:omNode-1 is not the leader. Suggested leader is OM:omNode-2[localhost/127.0.0.1]. 
      4507 at org.apache.hadoop.ozone.om.OzoneManager.checkLeaderStatus(OzoneManager.java:4014) 
      4508 at org.apache.hadoop.ozone.om.ratis.utils.OzoneManagerRatisUtils.checkLeaderStatus(OzoneManagerRatisUtils.java:473) 
      4509 at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:185) 
      4510 at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87) 
      4511 at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:148) 
      4512 at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java) 
      4513 at org.apache.hadoop.ipc.ProtobufRpcEngine$Server.processCall(ProtobufRpcEngine.java:484) 
      4514 at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:595) 
      4515 at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:573) 
      4516 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1227) 
      4517 at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1094) 
      4518 at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1017) 
      4519 at java.security.AccessController.doPrivileged(Native Method) 
      4520 at javax.security.auth.Subject.doAs(Subject.java:422) 
      4521 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) 
      4522 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3048) 
      4523
      4524 at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1567) 
      4525 at org.apache.hadoop.ipc.Client.call(Client.java:1513) 
      4526 at org.apache.hadoop.ipc.Client.call(Client.java:1410) 
      4527 at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:250) 
      4528 at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:132) 
      4529 at com.sun.proxy.$Proxy56.submitRequest(Unknown Source) 
      4530 at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source) 
      4531 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
      4532 at java.lang.reflect.Method.invoke(Method.java:498) 
      4533 at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:433) 
      4534 at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:166) 
      4535 at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:158) 
      4536 at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:96) 
      4537 at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:362) 
      4538 at com.sun.proxy.$Proxy56.submitRequest(Unknown Source) 
      4539 at org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransport.submitRequest(Hadoop3OmTransport.java:80) 
      4540 at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:322) 
      4541 at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.createVolume(OzoneManagerProtocolClientSideTranslatorPB.java:344) 
      4542 at org.apache.hadoop.ozone.client.rpc.RpcClient.createVolume(RpcClient.java:465) 
      4543 at org.apache.hadoop.ozone.client.ObjectStore.createVolume(ObjectStore.java:128) 
      4544 at org.apache.hadoop.ozone.om.TestOzoneManagerHAMetadataOnly.createAndCheckVolume(TestOzoneManagerHAMetadataOnly.java:87) 
      4545 at org.apache.hadoop.ozone.om.TestOzoneManagerHAMetadataOnly.testAllVolumeOperations(TestOzoneManagerHAMetadataOnly.java:103)
       

       

      https://github.com/apache/ozone/pull/5329

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            arafat2198 Arafat Khan
            arafat2198 Arafat Khan
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment