[HADOOP-16119] KMS on Hadoop RPC Engine

• Type: New Feature
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Assignee: Wei-Chiu Chuang
• Reporter: Jonathan Turner Eagles
• Created: 18/Feb/19 21:30
• Updated: 02/Oct/19 17:15

Description

Per discussion on common-dev and text copied here for ease of reference.

https://lists.apache.org/thread.html/0e2eeaf07b013f17fad6d362393f53d52041828feec53dcddff04808@%3Ccommon-dev.hadoop.apache.org%3E

Thanks all for the inputs,

To offer additional information (while Daryn is working on his stuff),
optimizing RPC encryption opens up another possibility: migrating KMS
service to use Hadoop RPC.

Today's KMS uses HTTPS + REST API, much like webhdfs. It has very
undesirable performance (a few thousand ops per second) compared to
NameNode. Unfortunately for each NameNode namespace operation you also need
to access KMS too.

Migrating KMS to Hadoop RPC greatly improves its performance (if
implemented correctly), and RPC encryption would be a prerequisite. So
please keep that in mind when discussing the Hadoop RPC encryption
improvements. Cloudera is very interested to help with the Hadoop RPC
encryption project because a lot of our customers are using at-rest
encryption, and some of them are starting to hit KMS performance limit.

This whole "migrating KMS to Hadoop RPC" was Daryn's idea. I heard this
idea in the meetup and I am very thrilled to see this happening because it
is a real issue bothering some of our customers, and I suspect it is the
right solution to address this tech debt.

Issue Links

• is blocked by: HADOOP-15977 RPC support for TLS
• relates to: HDFS-14312 KMS-o-meter: Scale test KMS using kms audit log

Sub-Tasks

• Abstract transport layer implementation out of KMS: Open
• Add protobuf and associated helper classes/interfaces for KMS: Open
• Create a KeyManagerRpcServer for KMS: Open
• Create a Hadoop RPC based KMS client: Patch Available
• Support delegation token operations in KMS Benchmark: Open
• Support reencrypt in KMS Benchmark: Patch Available