Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-10149

Locality Level is ANY on "Details for Stage" WebUI page

    Details

    • Type: Bug
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.4.1
    • Fix Version/s: None
    • Component/s: Spark Core, Web UI
    • Labels:
      None

      Description

      Locality Level is ANY on "Details for Stage" WebUI page

      When a sc.textFile(XX) program is running, Locality Level is ANY which should be NODE_LOCAL on stage 0.

      org.apache.spark.scheduler.TaskSetManager

      // Check for node-local tasks
      if (TaskLocality.isAllowed(locality, TaskLocality.NODE_LOCAL)) {
      for (index <- speculatableTasks if canRunOnHost(index)) {
      val locations = tasks(index).preferredLocations.map(_.host)
      if (locations.contains(host))

      Unknown macro: { speculatableTasks -= index return Some((index, TaskLocality.NODE_LOCAL)) }

      }
      }

      The variable "locations" is hostname of HDFS split, which is from InetAddress.getHostName.
      The variable "host" is ip of Executor, which is from InetAddress.getLocalHost.getHostAddress.

      org.apache.spark.deploy.worker.WorkerArguments

      var host = Utils.localHostName()

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                Yun Zhao Yun Zhao
              • Votes:
                0 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated: