[HDDS-7391] Phase I - Automated live rotation of CA certificates in a cluster with established trust

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: Security
• Labels: non_1.4.0_blocker pki
• Assignee: István Fajth
• Reporter: István Fajth
• Created: 24/Oct/22 11:01
• Updated: 21/Dec/23 19:33
• Resolved: 11/Dec/23 11:29

Description

The current rootCA certificate expiration happens in somewhat over 5 years after the certificate was created.
This event invalidates all certificates that are signed in the trust chain for which the rootCA certificate is the base of trust, this means that rotation and renewal of this certificate is time consuming at once, as it includes the renewal of all certificates.

In order to renew the rootCA certificate, instead of a full security re-bootstrap we would like to follow the following procedure:

• before the rootCA certificate expires, we create a new rootCA certificate
• with the new rootCA certificate we rotate the sub-CA certificate of all 3 SCMs
• once that is done, we make the new rootCA certificate available for other services via an SCM API
• other services are starting to poll for the new rootCA certificate at a time when it is most likely already generated and available via the SCM API
• once the new rootCA certificate is present, services update their TrustStores and after a random delay that leaves room for most if not all of the other services to refresh their TrustStores, every service renews it own certificate regardless of expiration, and gets a new certificate signed by the new sub-CA certificate of the leader.

During this process the start for polling the rootCA certificate happens around the same time, but this is a short request and the response payload is the rootCA certificate only, so SCM might experience a short peak here so we might want to introduce a jitter for this if necessary.

During this process the issuance of new certificates is a resource intensive task on the leader SCM, so we definitely want to introduce a jitter in that, a configurable one, in order to be able to shorten this period for testing.

More information on the failure scenarios and the whole process can be found in the attached pdf document.

Issue Links

• blocks: HDDS-7331 Ozone PKI improvements
• contains: HDDS-752 Functionality to handle key rotation in SCM
• is blocked by: HDDS-9420 [Compatibility]Enabling GRPC encryption causes SCM startup failure.
• is blocked by: HDDS-9442 Token verification from OMs at DT renew happens in the wrong login context.
• links to: Low Level Design of CA Rotation in SCM - V2

Sub-Tasks

• Create branch for CA cert rotation development: Resolved
• Prepare CertificateClient to store more than one rootCA certificate: Resolved
• Add SCM HA aware Root CA certificate monitor task: Resolved
• Prepare TrustManagers to handle multiple root CA certficates: Resolved
• Add initialization logic in CertificateClient to handle more than one rootCA certificate: Resolved
• Implement a protocol call to get the rootCA from SCM: Resolved
• Integrate rootCA rotation related changes to OM: Resolved
• Integrate rootCA rotation related changes to DN: Resolved
• Integrate root CA rotation related changes to Recon: Resolved
• Prohibit SCM role change initiated by user during Root CA rotation: Resolved
• ServiceInfo object to handle multiple rootCA certificates: Resolved
• Handle multiple rootCA certificates in the ServiceInfo object on the client side.: Resolved
• Test cross compatibility of a secured cluster vs old client: Resolved
• Test cross compatibility of a secure client vs old cluster: Resolved
• Add cleanup logic to remove old CA related material: Resolved
• Provide way to delete on disk old root CA key and certificate: Resolved
• Rotate Root CA and Sub CA in SCM: Resolved
• Create scheduler to check for new root ca certificates: Resolved
• Fetch and save all root certificates during service's certificate rotation: Resolved
• Add RootCARotationPoller to CertClient: Resolved
• Add the ability to turn off automated rootCA and sub-CA rotation: Resolved
• Implement the ability to update the ServiceInfo object with the new rootCA: Resolved
• Shared mutable state causes rootca poller to miss updates: Resolved
• Intermittent failure in test-root-ca-rotation.sh due to null certId: Resolved
• Intermittent timeout waiting for certificate 4: Resolved
• Failed to Read data from previously created key: Resolved
• Fix RootCaRotationPoller not toggling rotation if previous rotation failed: Resolved
• [ozone-cert-rotation] rootCA configs are referred even when RootCA is disabled: Resolved
• rootCA configs should not be checked when RootCA is disabled: Resolved
• Block CSR request in SCM for "hdds.x509.rootca.certificate.polling.interval" time period: Resolved
• Fetch root CA certificate list during SCM startup: Resolved
• Use sequence ID for certificate serial ID: Resolved
• Consolidate CertificateClient handleCase handling: Resolved
• Handle trust chain changes in clients when rootCAs are rotated.: Resolved
• Enable Root CA auto rotation by default: Open