Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6606

Optimize HDFS Encrypted Transport performance

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.6.0
    • datanode, hdfs-client, security
    • None
    • Reviewed
    • HDFS now supports the option to configure AES encryption for block data transfer. AES offers improved cryptographic strength and performance over the prior options of 3DES and RC4.

    Description

      In HDFS-3637, atm added support for encrypting the DataTransferProtocol, it was a great work.
      It utilizes SASL Digest-MD5 mechanism (use Qop: auth-conf), it supports three security strength:

      • high 3des or rc4 (128bits)
      • medium des or rc4(56bits)
      • low rc4(40bits)

      3des and rc4 are slow, only tens of MB/s,
      http://www.javamex.com/tutorials/cryptography/ciphers.shtml
      http://www.cs.wustl.edu/~jain/cse567-06/ftp/encryption_perf/

      I will give more detailed performance data in future. Absolutely it’s bottleneck and will vastly affect the end to end performance.

      AES(Advanced Encryption Standard) is recommended as a replacement of DES, it’s more secure; with AES-NI support, the throughput can reach nearly 2GB/s, it won’t be the bottleneck any more, AES and CryptoCodec work is supported in HADOOP-10150, HADOOP-10603 and HADOOP-10693 (We may need to add a new mode support for AES).

      This JIRA will use AES with AES-NI support as encryption algorithm for DataTransferProtocol.

      Attachments

        1. OptimizeHdfsEncryptedTransportperformance.pdf
          316 kB
          Yi Liu
        2. HDFS-6606.009.patch
          47 kB
          Yi Liu
        3. HDFS-6606.008.patch
          47 kB
          Yi Liu
        4. HDFS-6606.007.patch
          46 kB
          Yi Liu
        5. HDFS-6606.006.patch
          45 kB
          Yi Liu
        6. HDFS-6606.005.patch
          45 kB
          Yi Liu
        7. HDFS-6606.004.patch
          45 kB
          Yi Liu
        8. HDFS-6606.003.patch
          42 kB
          Yi Liu
        9. HDFS-6606.002.patch
          41 kB
          Yi Liu
        10. HDFS-6606.001.patch
          39 kB
          Yi Liu

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-7313 Support optional configuration of AES cipher suite on DataTransferProtocol.

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-10768 Optimize Hadoop RPC encryption performance

          • Major - Major loss of function.
          • Patch Available
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-9899 The implication of auth-conf is not followed in optimized HDFS data transfer encryption

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. HBASE-16633 Optimize HBase RPC Encryption Performance

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HDFS-3637 Add support for encrypting the DataTransferProtocol

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-2856 Fix block protocol so that Datanodes don't require root or jsvc

          • Major - Major loss of function.
          • Closed

          Activity

            People

              hitliuyi Yi Liu
              hitliuyi Yi Liu
              Votes:
              0 Vote for this issue
              Watchers:
              24 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: