[HDDS-7700] Recon server is missing CA certificates

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 1.2.0
• Component/s: Ozone Recon
• Labels: pki
• Assignee: Ashish Kumar
• Reporter: Soumitra Sulav
• Created: 22/Dec/22 15:19
• Updated: 31/May/23 10:06
• Resolved: 31/May/23 10:06

Description

Issue :

Certificates are missing for the Ozone RECON server.

Neither it's listed in the ozone admin cert list nor we can find it in the certs directory.

The startup logs of the Recon service confirm that there was an issue with the certificate creation.

Error StackTrace :

2022-12-21 13:43:23,575 INFO org.apache.hadoop.ozone.recon.ReconServer: Recon login successful.
2022-12-21 13:43:23,575 INFO org.apache.hadoop.ozone.recon.ReconServer: ReconStorageConfig initialized.Initializing certificate.
2022-12-21 13:43:23,575 INFO org.apache.hadoop.ozone.recon.ReconServer: Initializing secure Recon.
2022-12-21 13:43:24,246 ERROR org.apache.hadoop.hdds.security.x509.certificate.client.ReconCertificateClient: Default certificate serial id is not set. Can't locate the default certificate for this client.
2022-12-21 13:43:24,247 INFO org.apache.hadoop.hdds.security.x509.certificate.client.ReconCertificateClient: Certificate client init case: 6
2022-12-21 13:43:24,248 INFO org.apache.hadoop.hdds.security.x509.certificate.client.ReconCertificateClient: Found private and public key but certificate is missing.
2022-12-21 13:43:24,359 INFO org.apache.hadoop.ozone.recon.ReconServer: Init response: RECOVER
2022-12-21 13:43:24,360 ERROR org.apache.hadoop.ozone.recon.ReconServer: Recon security initialization failed. Recon certificate is missing.
2022-12-21 13:43:24,361 ERROR org.apache.hadoop.ozone.recon.ReconServer: Error during initializing Recon certificate
java.lang.RuntimeException: Recon security initialization failed.
        at org.apache.hadoop.ozone.recon.ReconServer.initializeCertificateClient(ReconServer.java:204)
        at org.apache.hadoop.ozone.recon.ReconServer.call(ReconServer.java:125)
        at org.apache.hadoop.ozone.recon.ReconServer.call(ReconServer.java:71)
        at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
        at picocli.CommandLine.access$1300(CommandLine.java:145)
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
        at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
        at picocli.CommandLine.execute(CommandLine.java:2078)
        at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:100)
        at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:91)
        at org.apache.hadoop.ozone.recon.ReconServer.main(ReconServer.java:92)
2022-12-21 13:43:24,364 INFO org.apache.hadoop.ozone.recon.spi.impl.ReconDBProvider: Last known Recon DB : /var/lib/hadoop-ozone/recon/data/recon-container-key.db_1671547060037
2022-12-21 13:43:24,626 INFO org.apache.hadoop.ozone.recon.persistence.DefaultDataSourceProvider: JDBC Url for Recon : jdbc:derby:/var/lib/hadoop-ozone/recon/data/ozone_recon_derby.db 

Cluster Details :

1. ozone version

Using HDDS 1.2.0