[SPARK-6373] Add SSL/TLS for the Netty based BlockTransferService

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Auto Closed
• Affects Version/s: 1.2.1
• Component/s: Block Manager, Shuffle, Spark Core
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Jeffrey Turpin
• Created: 17/Mar/15 02:50
• Updated: 23/Aug/23 23:14
• Resolved: 06/Jun/19 13:58

Description

Add the ability to allow for secure communications (SSL/TLS) for the Netty based BlockTransferService and the ExternalShuffleClient. This ticket will hopefully start the conversation around potential designs... Below is a reference to a WIP prototype which implements this functionality (prototype)... I have attempted to disrupt as little code as possible and tried to follow the current code structure (for the most part) in the areas I modified. I also studied how Hadoop achieves encrypted shuffle (http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html)

https://github.com/turp1twin/spark/commit/024b559f27945eb63068d1badf7f82e4e7c3621c

Issue Links

• relates to: SPARK-6229 Support SASL encryption in network/common module
• links to: [Github] Pull Request #9416 (turp1twin)