Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-6606

Optimize HDFS Encrypted Transport performance


    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.6.0
    • Component/s: datanode, hdfs-client, security
    • Labels:
    • Target Version/s:
    • Hadoop Flags:
    • Release Note:
      HDFS now supports the option to configure AES encryption for block data transfer. AES offers improved cryptographic strength and performance over the prior options of 3DES and RC4.


      In HDFS-3637, Aaron T. Myers added support for encrypting the DataTransferProtocol, it was a great work.
      It utilizes SASL Digest-MD5 mechanism (use Qop: auth-conf), it supports three security strength:

      • high 3des or rc4 (128bits)
      • medium des or rc4(56bits)
      • low rc4(40bits)

      3des and rc4 are slow, only tens of MB/s,

      I will give more detailed performance data in future. Absolutely it’s bottleneck and will vastly affect the end to end performance.

      AES(Advanced Encryption Standard) is recommended as a replacement of DES, it’s more secure; with AES-NI support, the throughput can reach nearly 2GB/s, it won’t be the bottleneck any more, AES and CryptoCodec work is supported in HADOOP-10150, HADOOP-10603 and HADOOP-10693 (We may need to add a new mode support for AES).

      This JIRA will use AES with AES-NI support as encryption algorithm for DataTransferProtocol.

      1. OptimizeHdfsEncryptedTransportperformance.pdf
        316 kB
        Yi Liu
      2. HDFS-6606.009.patch
        47 kB
        Yi Liu
      3. HDFS-6606.008.patch
        47 kB
        Yi Liu
      4. HDFS-6606.007.patch
        46 kB
        Yi Liu
      5. HDFS-6606.006.patch
        45 kB
        Yi Liu
      6. HDFS-6606.005.patch
        45 kB
        Yi Liu
      7. HDFS-6606.004.patch
        45 kB
        Yi Liu
      8. HDFS-6606.003.patch
        42 kB
        Yi Liu
      9. HDFS-6606.002.patch
        41 kB
        Yi Liu
      10. HDFS-6606.001.patch
        39 kB
        Yi Liu

        Issue Links


          No work has yet been logged on this issue.


            • Assignee:
              Yi Liu
              Yi Liu
            • Votes:
              0 Vote for this issue
              20 Start watching this issue


              • Created: