[HDFS-11210] Enhance key rolling to guarantee new KeyVersion is returned from generateEncryptedKeys after a key is rolled

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.6.5
• Fix Version/s: 3.0.0-alpha4
• Component/s: encryption, kms
• Target Version/s: 3.0.0-alpha4
• Hadoop Flags: Reviewed

Release Note

<!-- markdown -->

An `invalidateCache` command has been added to the KMS.
The `rollNewVersion` semantics of the KMS has been improved so that after a key's version is rolled, `generateEncryptedKey` of that key guarantees to return the `EncryptedKeyVersion` based on the new key version.

• Assignee: Xiao Chen
• Reporter: Xiao Chen
• Created: 06/Dec/16 06:32
• Updated: 19/Oct/17 18:17
• Resolved: 08/Feb/17 04:44

Description

To support re-encrypting EDEK, we need to make sure after a key is rolled, no old version EDEKs are used anymore. This includes various caches when generating EDEK.
This is not true currently, simply because no such requirements / necessities before.

This includes

• Client Provider(s), and corresponding cache(s).
  When LoadBalancingKMSCP is used, we need to clear all KMSCPs.
• KMS server instance(s), and corresponding cache(s)
  When KMS HA is configured with multiple KMS instances, only 1 will receive the rollNewVersion request, we need to make sure other instances are rolled too.
• The Client instance inside NN(s), and corresponding cache(s)
  When hadoop key roll is succeeded, the client provider inside NN should be drained too.

Issue Links

• breaks: HDFS-12667 KMSClientProvider#ValueQueue does synchronous fetch of edeks in background async thread.
• is depended upon by: HDFS-10899 Add functionality to re-encrypt EDEKs
• is duplicated by: HDFS-6971 Bounded staleness of EDEK caches on the NN