Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-6606

Optimize HDFS Encrypted Transport performance

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.6.0
    • Component/s: datanode, hdfs-client, security
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      HDFS now supports the option to configure AES encryption for block data transfer. AES offers improved cryptographic strength and performance over the prior options of 3DES and RC4.

      Description

      In HDFS-3637, Aaron T. Myers added support for encrypting the DataTransferProtocol, it was a great work.
      It utilizes SASL Digest-MD5 mechanism (use Qop: auth-conf), it supports three security strength:

      • high 3des or rc4 (128bits)
      • medium des or rc4(56bits)
      • low rc4(40bits)

      3des and rc4 are slow, only tens of MB/s,
      http://www.javamex.com/tutorials/cryptography/ciphers.shtml
      http://www.cs.wustl.edu/~jain/cse567-06/ftp/encryption_perf/

      I will give more detailed performance data in future. Absolutely it’s bottleneck and will vastly affect the end to end performance.

      AES(Advanced Encryption Standard) is recommended as a replacement of DES, it’s more secure; with AES-NI support, the throughput can reach nearly 2GB/s, it won’t be the bottleneck any more, AES and CryptoCodec work is supported in HADOOP-10150, HADOOP-10603 and HADOOP-10693 (We may need to add a new mode support for AES).

      This JIRA will use AES with AES-NI support as encryption algorithm for DataTransferProtocol.

      1. OptimizeHdfsEncryptedTransportperformance.pdf
        316 kB
        Yi Liu
      2. HDFS-6606.009.patch
        47 kB
        Yi Liu
      3. HDFS-6606.008.patch
        47 kB
        Yi Liu
      4. HDFS-6606.007.patch
        46 kB
        Yi Liu
      5. HDFS-6606.006.patch
        45 kB
        Yi Liu
      6. HDFS-6606.005.patch
        45 kB
        Yi Liu
      7. HDFS-6606.004.patch
        45 kB
        Yi Liu
      8. HDFS-6606.003.patch
        42 kB
        Yi Liu
      9. HDFS-6606.002.patch
        41 kB
        Yi Liu
      10. HDFS-6606.001.patch
        39 kB
        Yi Liu

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Yi Liu
              Reporter:
              Yi Liu
            • Votes:
              0 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development