Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6606

Optimize HDFS Encrypted Transport performance

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.6.0
    • datanode, hdfs-client, security
    • None
    • Reviewed
    • HDFS now supports the option to configure AES encryption for block data transfer. AES offers improved cryptographic strength and performance over the prior options of 3DES and RC4.

    Description

      In HDFS-3637, atm added support for encrypting the DataTransferProtocol, it was a great work.
      It utilizes SASL Digest-MD5 mechanism (use Qop: auth-conf), it supports three security strength:

      • high 3des or rc4 (128bits)
      • medium des or rc4(56bits)
      • low rc4(40bits)

      3des and rc4 are slow, only tens of MB/s,
      http://www.javamex.com/tutorials/cryptography/ciphers.shtml
      http://www.cs.wustl.edu/~jain/cse567-06/ftp/encryption_perf/

      I will give more detailed performance data in future. Absolutely it’s bottleneck and will vastly affect the end to end performance.

      AES(Advanced Encryption Standard) is recommended as a replacement of DES, it’s more secure; with AES-NI support, the throughput can reach nearly 2GB/s, it won’t be the bottleneck any more, AES and CryptoCodec work is supported in HADOOP-10150, HADOOP-10603 and HADOOP-10693 (We may need to add a new mode support for AES).

      This JIRA will use AES with AES-NI support as encryption algorithm for DataTransferProtocol.

      Attachments

        1. OptimizeHdfsEncryptedTransportperformance.pdf
          316 kB
          Yi Liu
        2. HDFS-6606.001.patch
          39 kB
          Yi Liu
        3. HDFS-6606.002.patch
          41 kB
          Yi Liu
        4. HDFS-6606.003.patch
          42 kB
          Yi Liu
        5. HDFS-6606.004.patch
          45 kB
          Yi Liu
        6. HDFS-6606.005.patch
          45 kB
          Yi Liu
        7. HDFS-6606.006.patch
          45 kB
          Yi Liu
        8. HDFS-6606.007.patch
          46 kB
          Yi Liu
        9. HDFS-6606.008.patch
          47 kB
          Yi Liu
        10. HDFS-6606.009.patch
          47 kB
          Yi Liu

        Issue Links

          Activity

            People

              hitliuyi Yi Liu
              hitliuyi Yi Liu
              Votes:
              0 Vote for this issue
              Watchers:
              24 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: