[SPARK-20531] Spark master shouldn't send its address back to the workers over proxied connections

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.6.3, 2.0.2, 2.1.2, 2.2.0
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Sameer Agarwal
• Created: 29/Apr/17 01:15
• Updated: 02/May/17 20:42
• Resolved: 02/May/17 20:42

Description

Currently, when a Spark worker connects to Spark master, the master sends its address back to the worker (as part of the RegisteredWorker message). The worker then saves this address and use it to talk to the master. The reason behind this handshake protocol is that if the master goes down, a new master can always send back RegisteredWorker messages to all the workers with its (new) IP address.

However, if there is a proxy between the master and worker, this unfortunately ends up bypassing the proxy. A simple fix here is that we should encode the "destination address" in the RegisterWorker that can then be sent back to the to the worker as part of the RegisteredWorker message.

cc zsxwing

Issue Links

• duplicates: SPARK-20529 Worker should not use the received Master address