[HADOOP-16050] S3A SSL connections should use OpenSSL

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.9.1
• Fix Version/s: 3.3.0
• Component/s: fs/s3
• Assignee: Sahil Takiar
• Reporter: Justin Uang
• Created: 17/Jan/19 20:18
• Updated: 21/Jan/20 16:39
• Resolved: 16/May/19 15:30

Description

We have found that when running the S3AFileSystem, it picks GCM as the ssl cipher suite. Unfortunately this is well known to be slow on java 8: https://stackoverflow.com/questions/25992131/slow-aes-gcm-encryption-and-decryption-with-java-8u20.

 

In practice we have seen that it can take well over 50% of our CPU time in spark workflows. We should add an option to set the list of cipher suites we would like to use.

Issue Links

• breaks: HADOOP-16321 ITestS3ASSL+TestOpenSSLSocketFactory failing with java.lang.UnsatisfiedLinkErrors
• causes: HADOOP-16347 MapReduce job tasks fails on S3A ssl3_get_server_certificate:certificate verify
• is related to: HADOOP-16346 Stabilize S3A OpenSSL support
• is related to: HADOOP-16405 Upgrade Wildfly Openssl version to 1.0.7.Final
• relates to: HADOOP-15669 ABFS: Improve HTTPS Performance
• relates to: HADOOP-16371 Option to disable GCM for SSL connections when running on Java 8
• relates to: IMPALA-8577 Crash during OpenSSLSocket.read
• relates to: HADOOP-15965 Upgrade to ADLS SDK which has major performance improvement for ingress/egress
• links to: GitHub Pull Request #784