[SPARK-4799] Spark should not rely on local host being resolvable on every node

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.1.0
• Component/s: Spark Core

Environment

Tested a Spark+Mesos cluster on top of Docker to reproduce the issue.

• Assignee: Unassigned
• Reporter: Santiago M. Mola
• Created: 09/Dec/14 17:37
• Updated: 01/Apr/15 11:00
• Resolved: 01/Apr/15 11:00

Description

Spark fails when a node hostname is not resolvable by other nodes.

See an example trace:

14/12/09 17:02:41 ERROR SendingConnection: Error connecting to 27e434cf36ac:35093
java.nio.channels.UnresolvedAddressException
	at sun.nio.ch.Net.checkAddress(Net.java:127)
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:644)
	at org.apache.spark.network.SendingConnection.connect(Connection.scala:299)
	at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:278)
	at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)

The relevant code is here:
https://github.com/apache/spark/blob/bcb5cdad614d4fce43725dfec3ce88172d2f8c11/core/src/main/scala/org/apache/spark/network/nio/ConnectionManager.scala#L170

val id = new ConnectionManagerId(Utils.localHostName, serverChannel.socket.getLocalPort)

This piece of code should use the host IP with Utils.localIpAddress or a method that acknowleges user settings (e.g. SPARK_LOCAL_IP). Since I cannot think about a use case for using hostname here, I'm creating a PR with the former solution, but if you think the later is better, I'm willing to create a new PR with a more elaborate fix.

Issue Links

• duplicates: SPARK-5078 Allow setting Akka host name from env vars
• is related to: SPARK-5113 Audit and document use of hostnames and IP addresses in Spark
• links to: GitHub Pull Request 3645