[HDDS-9444] S3SecretManager fails to revoke secret due to inconsistency between S3Cache and S3SecretTable.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: S3
• Labels: pull-request-available
• Assignee: Arafat Khan
• Reporter: Arafat Khan
• Created: 12/Oct/23 16:13
• Updated: 25/Jan/24 07:46
• Resolved: 22/Nov/23 01:25

Description

During a recent CI run, the TestSecureOzoneCluster test, specifically testGetSetRevokeS3Secret, failed. The issue encountered was that the S3 secret was not being revoked as expected. Upon examining the logs, it was determined that the discrepancy arose due to a mismatch between the value in the cache and the corresponding value in the database.

*CI Run :- https://github.com/ArafatKhan2198/ozone/actions/runs/6468031778/job/17559261272*#step:5:3776

Error Output :- 

Error:  Tests run: 13, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 120.823 s <<< FAILURE! - in org.apache.hadoop.ozone.TestSecureOzoneCluster
Error:  org.apache.hadoop.ozone.TestSecureOzoneCluster.testGetSetRevokeS3Secret  Time elapsed: 10.03 s  <<< ERROR!
S3_SECRET_ALREADY_EXISTS org.apache.hadoop.ozone.om.exceptions.OMException: Secret for 'om/fv[e-mail]' already exists
	at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:729)
	at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getS3Secret(OzoneManagerProtocolClientSideTranslatorPB.java:1017)
	at org.apache.hadoop.ozone.TestSecureOzoneCluster.testGetSetRevokeS3Secret(TestSecureOzoneCluster.java:740)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
	at org.junit.jupiter.engine.extension.TimeoutInvocation.proceed(TimeoutInvocation.java:46)
	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140) 

 
 

Issue Links

• fixes: HDDS-9349 Intermittent failure in TestSecureOzoneCluster#testGetSetRevokeS3Secret due to S3_SECRET_ALREADY_EXISTS
• relates to: HDDS-10200 OM may terminate due to NPE in S3SecretValue proto conversion
• links to: GitHub Pull Request #5445