[HDDS-6685] Follower OM crashed when validating S3 auth info.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Fix Version/s: 1.3.0
• Labels: pull-request-available
• Assignee: Sammi Chen
• Reporter: Sammi Chen
• Created: 02/May/22 03:27
• Updated: 21/Jan/24 19:41
• Resolved: 05/May/22 03:23

Description

The root cause of this issue is omRpcServer in OzoneManager holds a reference to delegationTokenMgr. 

Every incoming RPC request, omRpcServer will call delegationTokenMrg to validate the S3AuthInfo(All customer data are ingested using S3G) before checking the leadership of this OM instance.

During installCheckpoint, new metadataManager and delegationTokenMgr instances are created while omRpcServer still hold the old delegationTokenMgr reference.

 

So to make a clean context,  the solution might be stop the omRpcServer before the metadataManager stop. After checkpoint is installed, recreate metadataManager, delegationTokenMgr and then start a new omRpcServer server.  But this solution will cause IOException when leader OM or client(S3G) tries to send requests to this OM. Not sure how big the impact will be.

 

 

 

hs_err_pid716.log

Issue Links

• relates to: HDDS-10177 OM RPC server restarted by InstallSnapshotThread during shutdown
• relates to: HDDS-6763 SCM crashed while getting certificate from SCMCertStore
• links to: GitHub Pull Request #3376