[HDDS-9410] Upgrade stall from 1.3 to current code when gRPC TLS is enabled.

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Duplicate
• Affects Version/s: 1.4.0
• Component/s: Security
• Target Version/s: 1.4.0
• Assignee: István Fajth
• Reporter: István Fajth
• Created: 06/Oct/23 18:32
• Updated: 18/Oct/23 14:51
• Resolved: 18/Oct/23 14:51

Description

When a cluster is upgraded from the 1.3 release to the new code, with SCM HA enabled, and gRPC TLS turned on, SCM's stall and can not create the Ratis ring when starting up with the new code.

It is happening because during the Ratis server setup, ReloadingX509(Key|Trust)Manager tries to create the CertPath object that is used in the new code to identify a role, and the cert path creation tries to reach the SCM leader to get the CA certificates in the system, which fails and is being retried indefinitely, with that SCM remains stuck in this retry loop, as it is still in the Ratis ring creation.

A possible workaround for the problem is to concatenate the rootCA certificate to the certificate.crt and the <certSerialID>.crt file in the certificate directory (<ozone.metadata.dirs>/scm/sub-CA/certs/ folder).

Issue Links

• duplicates: HDDS-9420 [Compatibility]Enabling GRPC encryption causes SCM startup failure.
• is related to: HDDS-7379 Use certificate bundles instead of the sole certificate
• is related to: HDDS-7486 Support KeyStoreFactory which supports keyManager and trustManager reload
• relates to: HDDS-9420 [Compatibility]Enabling GRPC encryption causes SCM startup failure.