Details
-
Sub-task
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
None
-
None
Description
Some old clusters may have been switched from non-HA mode to HA mode,
and leave keys with higher updateID in the open key table.
It will cause OM to fail when assigning a lower updateID to OmKeyInfo, for example:
java.lang.IllegalArgumentException: Trying to set updateD to 2335683 which is not greater than the current value of 2974142 for OMKeyInfo{volume='s325d55ad283aa400af464c76d713c07ad', bucke t='idex-bucket',key='abcdef/20201102/plc_1604302955503_1511/0' , dataSize='12', creationTime='1604314644177', objectID='737325824', parentID='O', replication='RATIS/THREE', fileChecksu m='null} at org.apache.hadoop.ozone.om.helpers.WithObjectID.setUpdateID(WithObjectID.java:105) at org.apache.hadoop. ozone. om. request. key.0MOpenKeysDeleteRequest. update0penKeyTableCache (0MOpenKeysDeleteRequest, java: 163) at org.apache.hadoop. ozone. om. request.key. 0MOpenKeysDeleteRequest. validateAndUpdateCache (OMOpenKeysDeleteRequest. java: 95) at org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:293) at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine,java:529) at org.apache.hadoop. ozone. om. ratis.OzoneManagerStateMachine. lambda$1 (OzoneManagerStateMachine. java: 324) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture,java:1590) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util. concurrent.ThreadPoolExecutor$Worker. run (ThreadPoolExecutor, java: 624) at java. lang.Thread. run (Thread. java: 748)
In this case, the OpenKeyDeleteRequest should just ignore those keys.
And eventually, those keys will be deleted when the updateID grows and become new enough.
Attachments
Issue Links
- relates to
-
HDDS-9342 OM restart failed due to transactionLogIndex smaller than current updateID
- Resolved
- links to
PR merged, close this. Thanks ckj!