Details

    • Sub-task
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • None
    • 1.3.0
    • None

    Description

      Some old clusters may have been switched from non-HA mode to HA mode,

      and leave keys with higher updateID in the open key table.

      It will cause OM to fail when assigning a lower updateID to OmKeyInfo, for example:

      java.lang.IllegalArgumentException: Trying to set updateD to 2335683 which is not greater than the current value of 2974142 for OMKeyInfo{volume='s325d55ad283aa400af464c76d713c07ad', bucke
      t='idex-bucket',key='abcdef/20201102/plc_1604302955503_1511/0'
      , dataSize='12', creationTime='1604314644177', objectID='737325824', parentID='O', replication='RATIS/THREE', fileChecksu
      m='null}
      at org.apache.hadoop.ozone.om.helpers.WithObjectID.setUpdateID(WithObjectID.java:105)
      at org.apache.hadoop. ozone. om. request. key.0MOpenKeysDeleteRequest. update0penKeyTableCache (0MOpenKeysDeleteRequest, java: 163)
      at org.apache.hadoop. ozone. om. request.key. 0MOpenKeysDeleteRequest. validateAndUpdateCache (OMOpenKeysDeleteRequest. java: 95)
      at org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:293)
      at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine,java:529)
      at org.apache.hadoop. ozone. om. ratis.OzoneManagerStateMachine. lambda$1 (OzoneManagerStateMachine. java: 324)
      at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture,java:1590)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util. concurrent.ThreadPoolExecutor$Worker. run (ThreadPoolExecutor, java: 624)
      at java. lang.Thread. run (Thread. java: 748)

       

      In this case, the OpenKeyDeleteRequest should just ignore those keys.

      And eventually, those keys will be deleted when the updateID grows and become new enough.

      Attachments

        Issue Links

          Activity

            micahzhao mingchao zhao added a comment -

            PR merged, close this. Thanks ckj!

            micahzhao mingchao zhao added a comment - PR merged, close this. Thanks ckj !

            People

              ckj Kaijie Chen
              ckj Kaijie Chen
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: