Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-16119

KMS on Hadoop RPC Engine

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Per discussion on common-dev and text copied here for ease of reference.

      https://lists.apache.org/thread.html/0e2eeaf07b013f17fad6d362393f53d52041828feec53dcddff04808@%3Ccommon-dev.hadoop.apache.org%3E

      Thanks all for the inputs,
      
      To offer additional information (while Daryn is working on his stuff),
      optimizing RPC encryption opens up another possibility: migrating KMS
      service to use Hadoop RPC.
      
      Today's KMS uses HTTPS + REST API, much like webhdfs. It has very
      undesirable performance (a few thousand ops per second) compared to
      NameNode. Unfortunately for each NameNode namespace operation you also need
      to access KMS too.
      
      Migrating KMS to Hadoop RPC greatly improves its performance (if
      implemented correctly), and RPC encryption would be a prerequisite. So
      please keep that in mind when discussing the Hadoop RPC encryption
      improvements. Cloudera is very interested to help with the Hadoop RPC
      encryption project because a lot of our customers are using at-rest
      encryption, and some of them are starting to hit KMS performance limit.
      
      This whole "migrating KMS to Hadoop RPC" was Daryn's idea. I heard this
      idea in the meetup and I am very thrilled to see this happening because it
      is a real issue bothering some of our customers, and I suspect it is the
      right solution to address this tech debt.
      

        Attachments

        1. Design doc_ KMS v2.pdf
          325 kB
          Wei-Chiu Chuang

          Issue Links

            Activity

              People

              • Assignee:
                jojochuang Wei-Chiu Chuang
                Reporter:
                jeagles Jonathan Eagles
              • Votes:
                0 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated: