[HDDS-6998] Ignore open keys with higher updateID in OpenKeyDeleteRequest

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.3.0
• Labels: pull-request-available
• Assignee: Kaijie Chen
• Reporter: Kaijie Chen
• Created: 13/Jul/22 08:02
• Updated: 01/Feb/24 18:38
• Resolved: 14/Jul/22 11:10

Description

Some old clusters may have been switched from non-HA mode to HA mode,

and leave keys with higher updateID in the open key table.

It will cause OM to fail when assigning a lower updateID to OmKeyInfo, for example:

java.lang.IllegalArgumentException: Trying to set updateD to 2335683 which is not greater than the current value of 2974142 for OMKeyInfo{volume='s325d55ad283aa400af464c76d713c07ad', bucke
t='idex-bucket',key='abcdef/20201102/plc_1604302955503_1511/0'
, dataSize='12', creationTime='1604314644177', objectID='737325824', parentID='O', replication='RATIS/THREE', fileChecksu
m='null}
at org.apache.hadoop.ozone.om.helpers.WithObjectID.setUpdateID(WithObjectID.java:105)
at org.apache.hadoop. ozone. om. request. key.0MOpenKeysDeleteRequest. update0penKeyTableCache (0MOpenKeysDeleteRequest, java: 163)
at org.apache.hadoop. ozone. om. request.key. 0MOpenKeysDeleteRequest. validateAndUpdateCache (OMOpenKeysDeleteRequest. java: 95)
at org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:293)
at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine,java:529)
at org.apache.hadoop. ozone. om. ratis.OzoneManagerStateMachine. lambda$1 (OzoneManagerStateMachine. java: 324)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture,java:1590)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util. concurrent.ThreadPoolExecutor$Worker. run (ThreadPoolExecutor, java: 624)
at java. lang.Thread. run (Thread. java: 748)

 

In this case, the OpenKeyDeleteRequest should just ignore those keys.

And eventually, those keys will be deleted when the updateID grows and become new enough.

Issue Links

• relates to: HDDS-9342 OM restart failed due to transactionLogIndex smaller than current updateID
• links to: GitHub Pull Request #3596