Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11210

Enhance key rolling to guarantee new KeyVersion is returned from generateEncryptedKeys after a key is rolled

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.5
    • Fix Version/s: 3.0.0-alpha4
    • Component/s: encryption, kms
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      <!-- markdown -->

      An `invalidateCache` command has been added to the KMS.
      The `rollNewVersion` semantics of the KMS has been improved so that after a key's version is rolled, `generateEncryptedKey` of that key guarantees to return the `EncryptedKeyVersion` based on the new key version.
      Show
      <!-- markdown --> An `invalidateCache` command has been added to the KMS. The `rollNewVersion` semantics of the KMS has been improved so that after a key's version is rolled, `generateEncryptedKey` of that key guarantees to return the `EncryptedKeyVersion` based on the new key version.

      Description

      To support re-encrypting EDEK, we need to make sure after a key is rolled, no old version EDEKs are used anymore. This includes various caches when generating EDEK.
      This is not true currently, simply because no such requirements / necessities before.

      This includes

      • Client Provider(s), and corresponding cache(s).
        When LoadBalancingKMSCP is used, we need to clear all KMSCPs.
      • KMS server instance(s), and corresponding cache(s)
        When KMS HA is configured with multiple KMS instances, only 1 will receive the rollNewVersion request, we need to make sure other instances are rolled too.
      • The Client instance inside NN(s), and corresponding cache(s)
        When hadoop key roll is succeeded, the client provider inside NN should be drained too.

        Attachments

        1. HDFS-11210.05.patch
          37 kB
          Xiao Chen
        2. HDFS-11210.04.patch
          37 kB
          Xiao Chen
        3. HDFS-11210.03.patch
          36 kB
          Xiao Chen
        4. HDFS-11210.02.patch
          34 kB
          Xiao Chen
        5. HDFS-11210.01.patch
          21 kB
          Xiao Chen

          Issue Links

            Activity

              People

              • Assignee:
                xiaochen Xiao Chen
                Reporter:
                xiaochen Xiao Chen
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: