[HDDS-7184] Timed out acquiring authorizer write lock issue even if no multi-tenancy request is in progress

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: Ozone Client, Ozone Manager
• Assignee: Duong
• Reporter: Umang
• Created: 29/Aug/22 07:47
• Updated: 31/Aug/23 16:21
• Resolved: 31/Aug/23 16:20

Description

While running the multitenancy tests,timed out acquiring authorizer write lock issue appears even though the previous multi-tenancy requests are completed.

 

2022-08-11 20:49:51,886|INFO|MainThread|ozone.py:2000 - assignuser_tenant()|Assigning user to the tenant hrt_1 2022-08-11 20:49:51,887|INFO|MainThread|machine.py:186 - run()||GUID=814c6d93-4c0b-4efd-a951-6a843ee63410|RUNNING: klist -k -t /home/hrt_qa/hadoopqa/keytabs/hrt_qa.headless.keytab | grep -v HTTP 2022-08-11 20:49:51,894|INFO|MainThread|machine.py:228 - run()||GUID=814c6d93-4c0b-4efd-a951-6a843ee63410|Exit Code: 0 2022-08-11 20:49:51,894|INFO|MainThread|machine.py:2127 - get_principal_from_user()|--- user principal is [e-mail] 2022-08-11 20:49:51,894|INFO|MainThread|machine.py:186 - run()||GUID=3f33be35-21ec-4be1-89ca-d82bb317c382|RUNNING: /opt/cloudera/parcels/CDH/bin/ozone tenant user assign hrt_1 -t=tenant50989 --om-service-id=ozone1 2022-08-11 20:49:54,889|INFO|MainThread|machine.py:201 - run()||GUID=3f33be35-21ec-4be1-89ca-d82bb317c382|TIMEOUT Timed out acquiring authorizer write lock. Another multi-tenancy request is in-progress. Try again later 2022-08-11 20:49:54,904|INFO|MainThread|machine.py:230 - run()||GUID=3f33be35-21ec-4be1-89ca-d82bb317c382|Exit Code: 255 2022-08-11 20:49:54,905|INFO|MainThread|machine.py:236 - run()|Command /opt/cloudera/parcels/CDH/bin/ozone tenant user assign hrt_1 -t=tenant50989 --om-service-id=ozone1 failed after 0 retries 2022-08-11 20:49:54,905|INFO|MainThread|ozone.py:2106 - get_secret_format()|Getting secret for the tenant 2022-08-11 20:49:54,906|INFO|MainThread|machine.py:186 - run()||GUID=8e4b6662-b8ad-41e1-b1ca-6c7c0c6a8a74|RUNNING: klist -k -t /home/hrt_qa/hadoopqa/keytabs/hrt_qa.headless.keytab | grep -v HTTP 2022-08-11 20:49:54,913|INFO|MainThread|machine.py:228 - run()||GUID=8e4b6662-b8ad-41e1-b1ca-6c7c0c6a8a74|Exit Code: 0 2022-08-11 20:49:54,914|INFO|MainThread|machine.py:2127 - get_principal_from_user()|--- user principal is [e-mail] 2022-08-11 20:49:54,914|INFO|MainThread|machine.py:186 - run()||GUID=1c9fdb08-12da-428a-b793-f0926b87e4e5|RUNNING: /opt/cloudera/parcels/CDH/bin/ozone tenant user getsecret 'tenant50989$hrt_1' --om-service-id=ozone1 2022-08-11 20:49:56,939|INFO|MainThread|machine.py:201 - run()||GUID=1c9fdb08-12da-428a-b793-f0926b87e4e5|ACCESS_ID_NOT_FOUND accessId 'tenant50989$hrt_1' doesn't exist 2022-08-11 20:49:56,965|INFO|MainThread|machine.py:230 - run()||GUID=1c9fdb08-12da-428a-b793-f0926b87e4e5|Exit Code: 255 2022-08-11 20:49:56,965|INFO|MainThread|machine.py:236 - run()|Command /opt/cloudera/parcels/CDH/bin/ozone tenant user getsecret 'tenant50989$hrt_1' --om-service-id=ozone1 failed after 0 retries 2022-08-11 20:49:56,971|INFO|MainThread|conftest.py:348 - pytest_runtest_makereport()|call: <CallInfo when='call' exception: dictionary update sequence element #0 has length 1; 2 is required> 2022-08-11 20:49:56,971|INFO|MainThread|conftest.py:349 - pytest_runtest_makereport()|item: <Function 'test_deleteNonEmptyBucketWithVersions[FILE_SYSTEM_OPTIMIZED-True-False]'> 2022-08-11 20:49:57,090|INFO|MainThread|conftest.py:340 - pytest_report_teststatus()|TEST "test_deleteNonEmptyBucketWithVersions[FILE_SYSTEM_OPTIMIZED-True-False]" FAILED in 7.31 seconds

Issue Links

• is fixed by: HDDS-7755 S3G cannot acquire VOLUME_LOCK lock while holding [BUCKET_LOCK]