[HDDS-6427] OM Requests that depend on bucket layout should validate the layout of their bucket in validateAndUpdateCache

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Assignee: Jyotinder Singh
• Reporter: Ethan Rose
• Created: 07/Mar/22 23:38
• Updated: 22/Jun/22 12:06
• Resolved: 22/Jun/22 12:05

Description

Currently, FSO only requests (like OMKeyCommitRequestWithFSO), and requests that can be invoked on FSO buckets but will not work correctly (like OMKeysDeleteRequest and OMKeysRenameRequest) assume that they are being invoked on a bucket corresponding to the layout they are expecting.

For FSO-only requests, it is theoretically possible (though highly unlikely in practice) that they are invoked on a non-FSO bucket through the following race condition:

1. OMKeyRequestFactory#createRequest is invoked to determine the bucket layout. It determines an FSO request should be submitted.
2. Already in-flight bucket delete and create requests cause the original FSO bucket to be deleted, and a new one with the same name but different layout to take its place.
3. The FSO request is submitted to Ratis, and serialized to occur after the bucket create. When it is applied to the state machine, it is now operating on a non-FSO bucket.

There is a similar race if the bucket does not already exist, because OMKeyRequestFactory#createRequest returns the default bucket layout if the bucket is not found. By the time the transaction is applied to the state machine, an FSO bucket with this name may have already been created.

In general, any state machine data that is checked before validateAndUpdateCache must be considered stale when validateAndUpdateCache executes. It should be re-checked in validateAndUpdateCache to fail the request if the bucket type is incorrect.

 

For requests like OMKeysDeleteRequest and OMKeysRenameRequest which do not currently work with FSO, I am not sure what the plan is. If they are supposed to work with FSO, they should be updated as such (see HDDS-6414). If they are not supposed to work with FSO, they should return an error if invoked on an FSO bucket.