Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-10768

Optimize Hadoop RPC encryption performance

    Details

    • Type: Improvement
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: None
    • Component/s: performance, security
    • Labels:
      None
    • Release Note:
      Hide
      This improvement leverages AES-NI hardware acceleration to optimize Hadoop RPC performance.

      To enable it, adding the following configuration in core-site.xml
      <property>
      <name>hadoop.rpc.security.crypto.cipher.suites</name>
      <value>AES/CTR/NoPadding</value>
      </property>

      Optionally, set key bit length (default 128) via configuration key hadoop.rpc.security.crypto.cipher.key.bitlength in core-site.xml.
      Show
      This improvement leverages AES-NI hardware acceleration to optimize Hadoop RPC performance. To enable it, adding the following configuration in core-site.xml <property> <name>hadoop.rpc.security.crypto.cipher.suites</name> <value>AES/CTR/NoPadding</value> </property> Optionally, set key bit length (default 128) via configuration key hadoop.rpc.security.crypto.cipher.key.bitlength in core-site.xml.

      Description

      Hadoop RPC encryption is enabled by setting hadoop.rpc.protection to "privacy". It utilized SASL GSSAPI and DIGEST-MD5 mechanisms for secure authentication and data protection. Even GSSAPI supports using AES, but without AES-NI support by default, so the encryption is slow and will become bottleneck.

      After discuss with Aaron T. Myers, Alejandro Abdelnur and Uma Maheswara Rao G, we can do the same optimization as in HDFS-6606. Use AES-NI with more than 20x speedup.

      On the other hand, RPC message is small, but RPC is frequent and there may be lots of RPC calls in one connection, we needs to setup benchmark to see real improvement and then make a trade-off.

        Attachments

        1. cpu_profile_RPC_encryption_AES.png
          373 kB
          Wei-Chiu Chuang
        2. cpu_profile_rpc_encryption_optimize_calculateHMAC.png
          394 kB
          Wei-Chiu Chuang
        3. HADOOP-10768.001.patch
          71 kB
          Dian Fu
        4. HADOOP-10768.002.patch
          76 kB
          Dian Fu
        5. HADOOP-10768.003.patch
          78 kB
          Aaron T. Myers
        6. HADOOP-10768.004.patch
          121 kB
          Dapeng Sun
        7. HADOOP-10768.005.patch
          94 kB
          Dapeng Sun
        8. HADOOP-10768.006.patch
          95 kB
          Dapeng Sun
        9. HADOOP-10768.007.patch
          94 kB
          Dapeng Sun
        10. HADOOP-10768.008.patch
          94 kB
          Dapeng Sun
        11. HADOOP-10768.009.patch
          99 kB
          Wei-Chiu Chuang
        12. HADOOP-10768.010.patch
          95 kB
          Wei-Chiu Chuang
        13. HADOOP-10768.011.patch
          99 kB
          Wei-Chiu Chuang
        14. Optimize Hadoop RPC encryption performance.pdf
          280 kB
          Dian Fu

          Issue Links

            Activity

              People

              • Assignee:
                jojochuang Wei-Chiu Chuang
                Reporter:
                hitliuyi Yi Liu
              • Votes:
                0 Vote for this issue
                Watchers:
                53 Start watching this issue

                Dates

                • Created:
                  Updated: