[SPARK-10149] Locality Level is ANY on "Details for Stage" WebUI page

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 1.4.1
• Component/s: Spark Core, Web UI
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Yun Zhao
• Created: 21/Aug/15 07:03
• Updated: 21/May/19 04:15
• Resolved: 21/May/19 04:15

Description

Locality Level is ANY on "Details for Stage" WebUI page

When a sc.textFile(XX) program is running, Locality Level is ANY which should be NODE_LOCAL on stage 0.

org.apache.spark.scheduler.TaskSetManager

// Check for node-local tasks
if (TaskLocality.isAllowed(locality, TaskLocality.NODE_LOCAL)) {
for (index <- speculatableTasks if canRunOnHost(index)) {
val locations = tasks(index).preferredLocations.map(_.host)
if (locations.contains(host))

Unknown macro: { speculatableTasks -= index return Some((index, TaskLocality.NODE_LOCAL)) }

}
}

The variable "locations" is hostname of HDFS split, which is from InetAddress.getHostName.
The variable "host" is ip of Executor, which is from InetAddress.getLocalHost.getHostAddress.

org.apache.spark.deploy.worker.WorkerArguments

var host = Utils.localHostName()

Issue Links

• is duplicated by: SPARK-10006 Locality broken in spark 1.4.x for NewHadoopRDD
• is related to: SPARK-5113 Audit and document use of hostnames and IP addresses in Spark
• links to: [Github] Pull Request #8533 (wulei-bj-cn)