[HDDS-7332] Automatic OM/DN/Recon certificate rotation before certificate expiration

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: Security
• Labels: certificate_rotation pki
• Assignee: István Fajth
• Reporter: István Fajth
• Created: 14/Oct/22 21:16
• Updated: 04/Jun/24 08:42
• Resolved: 10/May/23 13:44

Description

As per the doc in HDDS-7331, the goals here are:

• implement a certificate owner driven certificate renewal before expiration in services
• implement certificate hotswap without service disruption
• introduce multiple certificates for different uses in services, separate these concerns on the certificates level
• start to include the whole trust chain in a certificate bundle, and use that instead of the sole certificate (with that allow us to have an arbitrary number of entities in the trust chain that we don't need to pre-distribute to truststores.)

Issue Links

• blocks: HDDS-7331 Ozone PKI improvements
• contains: HDDS-756 Functionality to handle key rotation in OM
• contains: HDDS-757 Functionality to handle key rotation in DN.
• is blocked by: RATIS-1747 Support keyManager and trustManager in tlsConfig

Sub-Tasks

• Check certificate expiration at service startup, renew if necessary: Resolved
• Implement Certificate renewal task for services: Resolved
• Support KeyStoreFactory which supports keyManager and trustManager reload: Resolved
• Use certificate bundles instead of the sole certificate: Resolved
• Ensure certificate hierarchy is set up properly: Resolved
• Upgrade ratis to 2.4.2-8b8bdda-SNAPSHOT: Resolved
• Incorrect synchronization during certificate renewal in DefaultCertificateClient: Resolved
• Certificate clients are not correctly closed.: Resolved
• Support fine grained certificate lifetime for efficient test: Resolved
• Use keyManager and trustManager provided by keyStoreFactory in datanode grpc services: Resolved
• Use keyManager and trustManager provided by keyStoreFactory in Ratis group: Resolved
• Use keyManager and trustManager provided by keyStoreFactory in om grpc services: Resolved
• Remove hadoop security dependency in Ozone org.apache.hadoop.hdds.security.ssl package: Resolved
• Refresh Keys and Certificate used in OzoneSecretManager after certificate renewed: Resolved
• Refactor the way to notify keyStoreFactory about certificate renewed: Resolved
• Support fine grained lifetime for root CA certificate: Resolved
• Validate that the public key belongs to the certificate during startup: Resolved
• Add cleanup in the certificate renewal logic to remove old pki material.: Resolved
• Remove the renew logic added by HDDS-7453: Resolved
• [ozone-cert-rotation][ozone] Multiple InternalCA were created: Resolved
• Refine certificate renewer service to avoid it scheduled ahead of time: Resolved