[SPARK-13317] SPARK_LOCAL_IP does not bind to public IP on Slaves

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Component/s: Deploy, EC2
• Labels: bulk-closed

Environment

Linux EC2, different VPC

• Assignee: Unassigned
• Reporter: Christopher Bourez
• Created: 14/Feb/16 16:36
• Updated: 21/May/19 04:32
• Resolved: 21/May/19 04:32

Description

SPARK_LOCAL_IP does not bind to the provided IP on slaves.
When launching a job or a spark-shell from a second network, the returned IP for the slave is still the first IP of the slave.
So the job fails with the message :

Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

It is not a question of resources but the driver which cannot connect to the slave given the wrong IP.