Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6606

Optimize HDFS Encrypted Transport performance

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.6.0
    • Component/s: datanode, hdfs-client, security
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      HDFS now supports the option to configure AES encryption for block data transfer. AES offers improved cryptographic strength and performance over the prior options of 3DES and RC4.

      Description

      In HDFS-3637, Aaron T. Myers added support for encrypting the DataTransferProtocol, it was a great work.
      It utilizes SASL Digest-MD5 mechanism (use Qop: auth-conf), it supports three security strength:

      • high 3des or rc4 (128bits)
      • medium des or rc4(56bits)
      • low rc4(40bits)

      3des and rc4 are slow, only tens of MB/s,
      http://www.javamex.com/tutorials/cryptography/ciphers.shtml
      http://www.cs.wustl.edu/~jain/cse567-06/ftp/encryption_perf/

      I will give more detailed performance data in future. Absolutely it’s bottleneck and will vastly affect the end to end performance.

      AES(Advanced Encryption Standard) is recommended as a replacement of DES, it’s more secure; with AES-NI support, the throughput can reach nearly 2GB/s, it won’t be the bottleneck any more, AES and CryptoCodec work is supported in HADOOP-10150, HADOOP-10603 and HADOOP-10693 (We may need to add a new mode support for AES).

      This JIRA will use AES with AES-NI support as encryption algorithm for DataTransferProtocol.

        Attachments

        1. HDFS-6606.001.patch
          39 kB
          Yi Liu
        2. HDFS-6606.002.patch
          41 kB
          Yi Liu
        3. HDFS-6606.003.patch
          42 kB
          Yi Liu
        4. HDFS-6606.004.patch
          45 kB
          Yi Liu
        5. HDFS-6606.005.patch
          45 kB
          Yi Liu
        6. HDFS-6606.006.patch
          45 kB
          Yi Liu
        7. HDFS-6606.007.patch
          46 kB
          Yi Liu
        8. HDFS-6606.008.patch
          47 kB
          Yi Liu
        9. HDFS-6606.009.patch
          47 kB
          Yi Liu
        10. OptimizeHdfsEncryptedTransportperformance.pdf
          316 kB
          Yi Liu

          Issue Links

            Activity

              People

              • Assignee:
                hitliuyi Yi Liu
                Reporter:
                hitliuyi Yi Liu
              • Votes:
                0 Vote for this issue
                Watchers:
                26 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: