[SPARK-11140] Replace file server in driver with RPC-based alternative

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.6.0, 2.0.0
• Component/s: Spark Core
• Assignee: Marcelo Masiero Vanzin
• Reporter: Marcelo Masiero Vanzin
• Created: 15/Oct/15 23:41
• Updated: 03/Dec/15 03:28
• Resolved: 23/Nov/15 21:54

Description

As part of making configuring encryption easy in Spark, it would be better to use the existing RPC channel between driver and executors to transfer files and jars added to the application.

This would remove the need to start the HTTP server currently used for that purpose, which needs to be configured to use SSL if encryption is wanted. SSL is kinda hard to configure correctly in a multi-user, distributed environment.

Issue Links

• blocks: SPARK-11563 Use RpcEnv to transfer generated classes in spark-shell
• is blocked by: SPARK-11235 Support streaming data using network library
• links to: [Github] Pull Request #9530 (vanzin)
• links to: [Github] Pull Request #9947 (vanzin)