Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-10768

Optimize Hadoop RPC encryption performance

    Details

    • Type: Improvement
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: None
    • Component/s: performance, security
    • Labels:
      None
    • Target Version/s:

      Description

      Hadoop RPC encryption is enabled by setting hadoop.rpc.protection to "privacy". It utilized SASL GSSAPI and DIGEST-MD5 mechanisms for secure authentication and data protection. Even GSSAPI supports using AES, but without AES-NI support by default, so the encryption is slow and will become bottleneck.

      After discuss with Aaron T. Myers, Alejandro Abdelnur and Uma Maheswara Rao G, we can do the same optimization as in HDFS-6606. Use AES-NI with more than 20x speedup.

      On the other hand, RPC message is small, but RPC is frequent and there may be lots of RPC calls in one connection, we needs to setup benchmark to see real improvement and then make a trade-off.

        Attachments

        1. HADOOP-10768.008.patch
          94 kB
          Dapeng Sun
        2. HADOOP-10768.007.patch
          94 kB
          Dapeng Sun
        3. HADOOP-10768.006.patch
          95 kB
          Dapeng Sun
        4. HADOOP-10768.005.patch
          94 kB
          Dapeng Sun
        5. HADOOP-10768.004.patch
          121 kB
          Dapeng Sun
        6. HADOOP-10768.003.patch
          78 kB
          Aaron T. Myers
        7. Optimize Hadoop RPC encryption performance.pdf
          280 kB
          Dian Fu
        8. HADOOP-10768.002.patch
          76 kB
          Dian Fu
        9. HADOOP-10768.001.patch
          71 kB
          Dian Fu

          Issue Links

            Activity

              People

              • Assignee:
                dapengsun Dapeng Sun
                Reporter:
                hitliuyi Yi Liu
              • Votes:
                0 Vote for this issue
                Watchers:
                43 Start watching this issue

                Dates

                • Created:
                  Updated: