[HDDS-7755] S3G cannot acquire VOLUME_LOCK lock while holding [BUCKET_LOCK]

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: SCM
• Labels: pull-request-available
• Target Version/s: 1.4.0, 1.3.1
• Assignee: Duong
• Reporter: Arun Sarin
• Created: 09/Jan/23 12:15
• Updated: 31/Aug/23 16:20
• Resolved: 20/Jan/23 15:52

Description

S3G cannot acquire VOLUME_LOCK lock while holding [BUCKET_LOCK].

Log from internal tests:

2022-12-09 12:57:32,982|INFO|MainThread|machine.py:230 - run()||GUID=39760bfb-bb7f-434e-8fa8-368701270fa3|Exit Code: 0 2022-12-09 12:57:32,982|INFO|MainThread|machine.py:188 - run()||GUID=c638945f-3903-4cec-81a9-cb15951608df|RUNNING: aws s3api --endpoint https://<clustername>:9879/ --ca-bundle=/usr/local/share/ca-certificates/ca.crt get-object --bucket erycnbhu --key file1 /tmp/getObjectFile1670590652 2022-12-09 12:57:33,475|INFO|MainThread|machine.py:203 - run()||GUID=c638945f-3903-4cec-81a9-cb15951608df| 2022-12-09 12:57:33,476|INFO|MainThread|machine.py:203 - run()||GUID=c638945f-3903-4cec-81a9-cb15951608df|An error occurred (NoSuchBucket) when calling the GetObject operation: The specified bucket does not exist 2022-12-09 12:57:33,547|INFO|MainThread|machine.py:232 - run()||GUID=c638945f-3903-4cec-81a9-cb15951608df|Exit Code: 255  

According to the locks priority mentioned in documentation 

https://ozone.apache.org/docs/1.0.0/design/locks.html

A higher priority lock in this case VOLUME_LOCK can't be acquired when we have lower priority lock i.e. BUCKET_LOCK

 

Issue Links

• fixes: HDDS-7184 Timed out acquiring authorizer write lock issue even if no multi-tenancy request is in progress
• is duplicated by: HDDS-8366 OzoneManager hangs when submitRequestToRatis
• is duplicated by: HDDS-7925 Deadlocks among all OMs in OM HA
• links to: GitHub Pull Request #4191