Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6606

Optimize HDFS Encrypted Transport performance

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments


    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.6.0
    • Component/s: datanode, hdfs-client, security
    • Labels:
    • Target Version/s:
    • Hadoop Flags:
    • Release Note:
      HDFS now supports the option to configure AES encryption for block data transfer. AES offers improved cryptographic strength and performance over the prior options of 3DES and RC4.


      In HDFS-3637, Aaron Myers added support for encrypting the DataTransferProtocol, it was a great work.
      It utilizes SASL Digest-MD5 mechanism (use Qop: auth-conf), it supports three security strength:

      • high 3des or rc4 (128bits)
      • medium des or rc4(56bits)
      • low rc4(40bits)

      3des and rc4 are slow, only tens of MB/s,

      I will give more detailed performance data in future. Absolutely it’s bottleneck and will vastly affect the end to end performance.

      AES(Advanced Encryption Standard) is recommended as a replacement of DES, it’s more secure; with AES-NI support, the throughput can reach nearly 2GB/s, it won’t be the bottleneck any more, AES and CryptoCodec work is supported in HADOOP-10150, HADOOP-10603 and HADOOP-10693 (We may need to add a new mode support for AES).

      This JIRA will use AES with AES-NI support as encryption algorithm for DataTransferProtocol.


        1. OptimizeHdfsEncryptedTransportperformance.pdf
          316 kB
          Yi Liu
        2. HDFS-6606.009.patch
          47 kB
          Yi Liu
        3. HDFS-6606.008.patch
          47 kB
          Yi Liu
        4. HDFS-6606.007.patch
          46 kB
          Yi Liu
        5. HDFS-6606.006.patch
          45 kB
          Yi Liu
        6. HDFS-6606.005.patch
          45 kB
          Yi Liu
        7. HDFS-6606.004.patch
          45 kB
          Yi Liu
        8. HDFS-6606.003.patch
          42 kB
          Yi Liu
        9. HDFS-6606.002.patch
          41 kB
          Yi Liu
        10. HDFS-6606.001.patch
          39 kB
          Yi Liu

        Issue Links


          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users



              • Created:

                Issue deployment