[HADOOP-16876] KMS delegation tokens are memory expensive

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Component/s: kms
• Assignee: Unassigned
• Reporter: Wei-Chiu Chuang
• Created: 21/Feb/20 01:13
• Updated: 23/Jan/23 16:37
• Resolved: 23/Jan/23 16:37

Description

We recently saw a number of users reporting high memory consumption in KMS.

Part of the reason being HADOOP-14445. Without that, the number of kms delegation tokens that zookeeper stores is proportional to the number of KMS servers.

There are two problems:
(1) it exceeds zookeeper jute buffer length and operations fail.
(2) KMS uses more heap memory to store KMS DTs.

But even with HADOOP-14445, KMS DTs are still expensive. Looking at a heap dump from KMS, the majority of the heap is occupied by znode and KMS DT objects. With the growing number of encrypted clusters and use cases, this is increasingly a problem our users encounter.

Issue Links

• relates to: HADOOP-16828 Zookeeper Delegation Token Manager fetch sequence number by batch
• relates to: HDFS-15383 RBF: Disable watch in ZKDelegationSecretManager for performance