Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-10768

Optimize Hadoop RPC encryption performance


    • Type: Improvement
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: None
    • Component/s: performance, security
    • Labels:
    • Target Version/s:


      Hadoop RPC encryption is enabled by setting hadoop.rpc.protection to "privacy". It utilized SASL GSSAPI and DIGEST-MD5 mechanisms for secure authentication and data protection. Even GSSAPI supports using AES, but without AES-NI support by default, so the encryption is slow and will become bottleneck.

      After discuss with Aaron T. Myers, Alejandro Abdelnur and Uma Maheswara Rao G, we can do the same optimization as in HDFS-6606. Use AES-NI with more than 20x speedup.

      On the other hand, RPC message is small, but RPC is frequent and there may be lots of RPC calls in one connection, we needs to setup benchmark to see real improvement and then make a trade-off.


        1. HADOOP-10768.008.patch
          94 kB
          Dapeng Sun
        2. HADOOP-10768.007.patch
          94 kB
          Dapeng Sun
        3. HADOOP-10768.006.patch
          95 kB
          Dapeng Sun
        4. HADOOP-10768.005.patch
          94 kB
          Dapeng Sun
        5. HADOOP-10768.004.patch
          121 kB
          Dapeng Sun
        6. HADOOP-10768.003.patch
          78 kB
          Aaron T. Myers
        7. Optimize Hadoop RPC encryption performance.pdf
          280 kB
          Dian Fu
        8. HADOOP-10768.002.patch
          76 kB
          Dian Fu
        9. HADOOP-10768.001.patch
          71 kB
          Dian Fu

          Issue Links



              • Assignee:
                dapengsun Dapeng Sun
                hitliuyi Yi Liu
              • Votes:
                0 Vote for this issue
                43 Start watching this issue


                • Created: