[SPARK-44937] Add SSL/TLS support for RPC and Shuffle communications

• Type: Epic
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Block Manager, Security, Shuffle, Spark Core
• Labels: releasenotes
• Epic Name: ssl/tls for rpc and shuffle
• Assignee: Hasnain Lakhani
• Reporter: Hasnain Lakhani
• Created: 23/Aug/23 23:12
• Updated: 30/Jul/24 04:39
• Resolved: 08/Nov/23 03:24

Description

Add support for SSL/TLS based communication for Spark RPCs and block transfers - providing an alternative to the existing encryption / authentication implementation documented at https://spark.apache.org/docs/latest/security.html#spark-rpc-communication-protocol-between-spark-processes

This is a superset of the functionality discussed in https://issues.apache.org/jira/browse/SPARK-6373

Issue Links

• is related to: SPARK-45376 Add netty-tcnative-boringssl-static dependency
• is related to: SPARK-44111 Prepare Apache Spark 4.0.0
• links to: GitHub Pull Request #42685
• links to: GitHub Pull Request #43162
• links to: GitHub Pull Request #43163
• links to: GitHub Pull Request #43164
• links to: GitHub Pull Request #43165
• links to: GitHub Pull Request #43166