Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-7332 Automatic OM/DN/Recon certificate rotation before certificate expiration
  3. HDDS-9042

[ozone-cert-rotation][ozone] Multiple InternalCA were created

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Critical
    • Resolution: Duplicate
    • 1.4.0
    • 1.4.0
    • Ozone Manager

    Description

      Below configs were used.

      <property><name>hdds.x509.max.duration</name><value>P5D</value></property><property><name>hdds.x509.default.duration</name><value>PT1H</value></property><property><name>hdds.x509.renew.grace.duration</name><value>PT50M</value></property><property><name>hdds.x509.ca.rotation.check.interval</name><value>PT30M</value></property><property><name>hdds.block.token.expiry.time</name><value>15m</value></property><property><name>ozone.manager.delegation.token.renew-interval</name><value>15m</value></property><property><name>ozone.manager.delegation.token.max-lifetime</name><value>30m</value></property>

      Based on the above config, every 10 minutes new config will be generated.
      Observation (E.g. om) :

      • There were some delays in a few of the certificate generation.
      • Multiple certificates generated at the same time with the same expiry and same CN (different Sl. No.)
      54179279105237591 Wed Jul 19 15:29:58 UTC 2023   Wed Jul 19 16:29:58 UTC 2023   CN=om@quasar-hkdejw-2.quasar-hkdejw.root.hwx.site,OU=45e34b55-8f7a-4cd6-8a3d-bd5eab2a77ea,O=CID-dc676dc7-03ab-4995-a62d-5846f54b2188 CN=scm-sub-54179249046570545@quasar-hkdejw-1.quasar-hkdejw.root.hwx.site,OU=45e34b55-8f7a-4cd6-8a3d-bd5eab2a77ea,O=CID-dc676dc7-03ab-4995-a62d-5846f54b2188
54179872955682725 Wed Jul 19 15:39:51 UTC 2023   Wed Jul 19 16:39:51 UTC 2023   CN=om@quasar-hkdejw-5.quasar-hkdejw.root.hwx.site,OU=45e34b55-8f7a-4cd6-8a3d-bd5eab2a77ea,O=CID-dc676dc7-03ab-4995-a62d-5846f54b2188 CN=scm-sub-54179249046570545@quasar-hkdejw-1.quasar-hkdejw.root.hwx.site,OU=45e34b55-8f7a-4cd6-8a3d-bd5eab2a77ea,O=CID-dc676dc7-03ab-4995-a62d-5846f54b2188
54179875311720227 Wed Jul 19 15:39:54 UTC 2023   Wed Jul 19 16:39:54 UTC 2023   CN=om@quasar-hkdejw-7.quasar-hkdejw.root.hwx.site,OU=45e34b55-8f7a-4cd6-8a3d-bd5eab2a77ea,O=CID-dc676dc7-03ab-4995-a62d-5846f54b2188 CN=scm-sub-54179249046570545@quasar-hkdejw-1.quasar-hkdejw.root.hwx.site,OU=45e34b55-8f7a-4cd6-8a3d-bd5eab2a77ea,O=CID-dc676dc7-03ab-4995-a62d-5846f54b2188
54179879754295550 Wed Jul 19 15:39:58 UTC 2023   Wed Jul 19 16:39:58 UTC 2023   CN=om@quasar-hkdejw-2.quasar-hkdejw.root.hwx.site,OU=45e34b55-8f7a-4cd6-8a3d-bd5eab2a77ea,O=CID-dc676dc7-03ab-4995-a62d-5846f54b2188 CN=scm-sub-54179249046570545@quasar-hkdejw-1.quasar-hkdejw.root.hwx.site,OU=45e34b55-8f7a-4cd6-8a3d-bd5eab2a77ea,O=CID-dc676dc7-03ab-4995-a62d-5846f54b2188
54180872472759626 Wed Jul 19 15:56:31 UTC 2023   Wed Jul 19 16:56:31 UTC 2023   CN=om@quasar-hkdejw-5.quasar-hkdejw.root.hwx.site,OU=45e34b55-8f7a-4cd6-8a3d-bd5eab2a77ea,O=CID-dc676dc7-03ab-4995-a62d-5846f54b2188 CN=scm-sub-54179249046570545@quasar-hkdejw-1.quasar-hkdejw.root.hwx.site,OU=45e34b55-8f7a-4cd6-8a3d-bd5eab2a77ea,O=CID-dc676dc7-03ab-4995-a62d-5846f54b2188
54180875391421145 Wed Jul 19 15:56:34 UTC 2023   Wed Jul 19 16:56:34 UTC 2023   CN=om@quasar-hkdejw-7.quasar-hkdejw.root.hwx.site,OU=45e34b55-8f7a-4cd6-8a3d-bd5eab2a77ea,O=CID-dc676dc7-03ab-4995-a62d-5846f54b2188 CN=scm-sub-54179249046570545@quasar-hkdejw-1.quasar-hkdejw.root.hwx.site,OU=45e34b55-8f7a-4cd6-8a3d-bd5eab2a77ea,O=CID-dc676dc7-03ab-4995-a62d-5846f54b2188
54180879428435773 Wed Jul 19 15:56:38 UTC 2023   Wed Jul 19 16:56:38 UTC 2023   CN=om@quasar-hkdejw-2.quasar-hkdejw.root.hwx.site,OU=45e34b55-8f7a-4cd6-8a3d-bd5eab2a77ea,O=CID-dc676dc7-03ab-4995-a62d-5846f54b2188 CN=scm-sub-54179249046570545@quasar-hkdejw-1.quasar-hkdejw.root.hwx.site,OU=45e34b55-8f7a-4cd6-8a3d-bd5eab2a77ea,O=CID-dc676dc7-03ab-4995-a62d-5846f54b2188
54181805046269983 Wed Jul 19 16:12:03 UTC 2023   Wed Jul 19 17:12:03 UTC 2023   CN=om@quasar-hkdejw-7.quasar-hkdejw.root.hwx.site,OU=45e34b55-8f7a-4cd6-8a3d-bd5eab2a77ea,O=CID-dc676dc7-03ab-4995-a62d-5846f54b2188 CN=scm-sub-54179249046570545@quasar-hkdejw-1.quasar-hkdejw.root.hwx.site,OU=45e34b55-8f7a-4cd6-8a3d-bd5eab2a77ea,O=CID-dc676dc7-03ab-4995-a62d-5846f54b2188
54181807590705776 Wed Jul 19 16:12:06 UTC 2023   Wed Jul 19 17:12:06 UTC 2023   CN=om@quasar-hkdejw-2.quasar-hkdejw.root.hwx.site,OU=45e34b55-8f7a-4cd6-8a3d-bd5eab2a77ea,O=CID-dc676dc7-03ab-4995-a62d-5846f54b2188 CN=scm-sub-54179249046570545@quasar-hkdejw-1.quasar-hkdejw.root.hwx.site,OU=45e34b55-8f7a-4cd6-8a3d-bd5eab2a77ea,O=CID-dc676dc7-03ab-4995-a62d-5846f54b2188
54181807732629049 Wed Jul 19 16:12:06 UTC 2023   Wed Jul 19 17:12:06 UTC 2023   CN=om@quasar-hkdejw-2.quasar-hkdejw.root.hwx.site,OU=45e34b55-8f7a-4cd6-8a3d-bd5eab2a77ea,O=CID-dc676dc7-03ab-4995-a62d-5846f54b2188 CN=scm-sub-54179249046570545@quasar-hkdejw-1.quasar-hkdejw.root.hwx.site,OU=45e34b55-8f7a-4cd6-8a3d-bd5eab2a77ea,O=CID-dc676dc7-03ab-4995-a62d-5846f54b2188
54181810223644744 Wed Jul 19 16:12:09 UTC 2023   Wed Jul 19 17:12:09 UTC 2023   CN=om@quasar-hkdejw-5.quasar-hkdejw.root.hwx.site,OU=45e34b55-8f7a-4cd6-8a3d-bd5eab2a77ea,O=CID-dc676dc7-03ab-4995-a62d-5846f54b2188 CN=scm-sub-54179249046570545@quasar-hkdejw-1.quasar-hkdejw.root.hwx.site,OU=45e34b55-8f7a-4cd6-8a3d-bd5eab2a77ea,O=CID-dc676dc7-03ab-4995-a62d-5846f54b2188
54181810247901998 Wed Jul 19 16:12:09 UTC 2023   Wed Jul 19 17:12:09 UTC 2023   CN=om@quasar-hkdejw-5.quasar-hkdejw.root.hwx.site,OU=45e34b55-8f7a-4cd6-8a3d-bd5eab2a77ea,O=CID-dc676dc7-03ab-4995-a62d-5846f54b2188 CN=scm-sub-54179249046570545@quasar-hkdejw-1.quasar-hkdejw.root.hwx.site,OU=45e34b55-8f7a-4cd6-8a3d-bd5eab2a77ea,O=CID-dc676dc7-03ab-4995-a62d-5846f54b2188

      Live Cluster stopped to preserve the state :
      https://quasar-hkdejw-1.quasar-hkdejw.root.hwx.site:7183/cmf/services/1546335626/config#q=safety

      Attachments

        Issue Links

          fixes

          Sub-task - The sub-task of the issue HDDS-9044 [ozone-cert-rotation] Invalid exit code from ozone cert clean command

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          is fixed by

          Sub-task - The sub-task of the issue HDDS-9066 Remove the renew logic added by HDDS-7453

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HDDS-9061 Possible dead-lock in SCM initialization due to certificate operations.

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          links to

          Web Link pull request for resolving the issue

          Activity

            People

              Unassigned Unassigned
              ssulav Soumitra Sulav
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: