Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-9342

OM restart failed due to transactionLogIndex smaller than current updateID

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Duplicate
    • 1.3.0
    • None
    • OM, OM HA

    Description

      OM restart failed, log as follow:
      create failed:

      java.lang.IllegalArgumentException: Trying to set updateID to 2901863625 which is not greater than the current value of 2901863627 for OMKeyInfo{volume='vol-xxx', bucket='xxx', key='user/xxx/platform/xxx', dataSize='268435456', creationTime='1695088210914', objectID='-9223371293977687808', parentID='0', replication='RATIS/THREE', fileChecksum='null}
              at org.apache.hadoop.ozone.om.helpers.WithObjectID.setUpdateID(WithObjectID.java:105)
              at org.apache.hadoop.ozone.om.request.key.OMKeyRequest.prepareFileInfo(OMKeyRequest.java:665)
              at org.apache.hadoop.ozone.om.request.key.OMKeyRequest.prepareKeyInfo(OMKeyRequest.java:623)
              at org.apache.hadoop.ozone.om.request.file.OMFileCreateRequest.validateAndUpdateCache(OMFileCreateRequest.java:255)
              at org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:311)
              at org.apache.hadoop.ozone.protocolPB.OzoneManagerRouterRequestHandler.handleWriteRequest(OzoneManagerRouterRequestHandler.java:806)
              at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:535)
              at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$1(OzoneManagerStateMachine.java:326)
              at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      
      

      rename failed:

      java.lang.IllegalArgumentException: Trying to set updateID to 2901863669 which is not greater than the current value of 3076345041 for OMKeyInfo{volume='vol-xxx', bucket='xxx', key='checkative/xxx', dataSize='23124', creationTime='1695380440059', objectID='-9223371249310446848', parentID='0', replication='RATIS/THREE', fileChecksum='null}
              at org.apache.hadoop.ozone.om.helpers.WithObjectID.setUpdateID(WithObjectID.java:105)
              at org.apache.hadoop.ozone.om.request.key.OMKeyRenameRequest.validateAndUpdateCache(OMKeyRenameRequest.java:190)
              at org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:311)
              at org.apache.hadoop.ozone.protocolPB.OzoneManagerRouterRequestHandler.handleWriteRequest(OzoneManagerRouterRequestHandler.java:806)
              at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:535)
              at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$1(OzoneManagerStateMachine.java:326)
              at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      

      Attachments

        1. clipboard_image_1700795744614.png
          1.12 MB
          guangbao zhao
        2. HDDS-9342_reproduce_20231217.patch
          8 kB
          Hongbing Wang
        3. HDDS-9342_testUpdateId_reproduce.patch
          4 kB
          Hongbing Wang
        4. HDDS-9342_testUpdateId.patch
          6 kB
          Tsz-wo Sze
        5. om.shutdown-20230922.log
          31 kB
          Hongbing Wang

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Sammi Sammi Chen
            wanghongbing Hongbing Wang
            Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment