[SPARK-2157] Can't write tight firewall rules for Spark

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.1.0
• Component/s: Deploy, Spark Core
• Target Version/s: 1.1.0
• Assignee: Andrew Ash
• Reporter: Andrew Ash
• Created: 16/Jun/14 21:24
• Updated: 13/Dec/14 19:36
• Resolved: 06/Aug/14 07:10

Description

In order to run Spark in places with strict firewall rules, you need to be able to specify every port that's used between all parts of the stack.

Per the network activity section of the docs most of the ports are configurable, but there are a few ports that aren't configurable.

We need to make every port configurable to a particular port, so that we can run Spark in highly locked-down environments.

Issue Links

• is blocked by: SPARK-1176 Adding port configuration for HttpBroadcast
• is blocked by: SPARK-1174 Adding port configuration for HttpFileServer
• relates to: SPARK-4837 NettyBlockTransferService does not abide by spark.blockManager.port config option
• relates to: SPARK-1174 Adding port configuration for HttpFileServer
• links to: [Github] Pull Request #1107 (ash211)
• links to: [Github] Pull Request #1777 (andrewor14)