Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-16017

YarnClientSchedulerBackend now registers backends as IPs instead of Hostnames which causes all tasks to run with RACK_LOCAL locality.

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 1.6.2, 2.0.0
    • 1.6.2, 2.0.0
    • Spark Core
    • None

    Description

      Since this change: SPARK-15395

      When registering new executor backends it registers them as IPs instead of hostnames. This causes a flow on effect that when the Task manager is trying to figure out what Locality tasks should run at, no tasks can be run At the NODE_LOCAL level.

      This specific call:
      https://github.com/apache/spark/blob/branch-2.0/core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala#L886

      pendingTasksForHost are all hostnames pulled from the DFS locations while hasExecutorsAliveOnHost, uses executorsByHost, which are all IP's because they are populated from the RpcAddress.

      As expected this causes significant performance problems, A simple count query will take 22 seconds, But if I revert the change from SPARK-15395, tasks will run with NODE_LOCAL locality and the same count will take 3 seconds.

      Attachments

        Issue Links

          Activity

            This is a pretty bad performance regression. zsxwing

            Very tempted to up it to blocker.

            vanzin Marcelo Masiero Vanzin added a comment - This is a pretty bad performance regression. zsxwing Very tempted to up it to blocker.

            BTW SPARK-15395 doesn't really explain a real use case where the old code caused a problem; given that we've had a few releases with that code and I've never seen any issues related to it, at this point I'd suggest just reverting that patch and, if there's a real issue, fixing it properly in the next point release.

            vanzin Marcelo Masiero Vanzin added a comment - BTW SPARK-15395 doesn't really explain a real use case where the old code caused a problem; given that we've had a few releases with that code and I've never seen any issues related to it, at this point I'd suggest just reverting that patch and, if there's a real issue, fixing it properly in the next point release.

            In fact this shouldn't affect just YARN, but anything that uses HDFS or any storage system that reports hostnames when providing info about splits.

            vanzin Marcelo Masiero Vanzin added a comment - In fact this shouldn't affect just YARN, but anything that uses HDFS or any storage system that reports hostnames when providing info about splits.
            zsxwing Shixiong Zhu added a comment -

            vanzin what do you think if we just send hostname from CoarseGrainedSchedulerBackend and set it in https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L167 ?

            SPARK-15395 is to improve the robustness. If there is no way to fix this one, I agree that we just revert SPARK-15395.

            zsxwing Shixiong Zhu added a comment - vanzin what do you think if we just send hostname from CoarseGrainedSchedulerBackend and set it in https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L167 ? SPARK-15395 is to improve the robustness. If there is no way to fix this one, I agree that we just revert SPARK-15395 .

            I assume you mean from CoarseGrainedExecutorBackend? That could work. You could also resolve the executor's IP in CoarseGrainedSchedulerBackend, although that might be slower.

            vanzin Marcelo Masiero Vanzin added a comment - I assume you mean from CoarseGrainedExecutorBackend ? That could work. You could also resolve the executor's IP in CoarseGrainedSchedulerBackend , although that might be slower.
            zsxwing Shixiong Zhu added a comment -

            Yes. Thanks for correcting. I will submit a PR for 2.0. In the mean time, let's revert my patch for batch 1.6 as it seems too much changes for 1.6.

            zsxwing Shixiong Zhu added a comment - Yes. Thanks for correcting. I will submit a PR for 2.0. In the mean time, let's revert my patch for batch 1.6 as it seems too much changes for 1.6.
            zsxwing Shixiong Zhu added a comment -

            vanzin Just realized one problem: If the user starts the executors using IP addresses, we will still report IPs to TaskScheduler. vanzin do you know how HDFS reports hostname? Will it reports hostname even if the user starts the datanodes using IP addresses? If so, my proposal won't fix it and I think "resolving the executor's IP in CoarseGrainedSchedulerBackend" is better.

            zsxwing Shixiong Zhu added a comment - vanzin Just realized one problem: If the user starts the executors using IP addresses, we will still report IPs to TaskScheduler. vanzin do you know how HDFS reports hostname? Will it reports hostname even if the user starts the datanodes using IP addresses? If so, my proposal won't fix it and I think "resolving the executor's IP in CoarseGrainedSchedulerBackend" is better.

            The default configuration of HDFS seems to disallow the case where the DN connects just with an IP address - the hostname should be resolved. So I think it's pretty sure to assume that you'll always have hostnames.

            (That code is in HDFS's DatanodeManager.java, btw.)

            vanzin Marcelo Masiero Vanzin added a comment - The default configuration of HDFS seems to disallow the case where the DN connects just with an IP address - the hostname should be resolved. So I think it's pretty sure to assume that you'll always have hostnames. (That code is in HDFS's DatanodeManager.java, btw.)
            apachespark Apache Spark added a comment -

            User 'zsxwing' has created a pull request for this issue:
            https://github.com/apache/spark/pull/13741

            apachespark Apache Spark added a comment - User 'zsxwing' has created a pull request for this issue: https://github.com/apache/spark/pull/13741
            srowen Sean R. Owen added a comment -

            BTW while we're here... is this related to what https://github.com/apache/spark/pull/8533 was talking about? I know, this is narrowly about SPARK-15395 but wondering if it is something that is or was already resolved.

            srowen Sean R. Owen added a comment - BTW while we're here... is this related to what https://github.com/apache/spark/pull/8533 was talking about? I know, this is narrowly about SPARK-15395 but wondering if it is something that is or was already resolved.
            zsxwing Shixiong Zhu added a comment -

            Just to confirm that: you agree that just sending the hostname from executors to driver. Right?

            zsxwing Shixiong Zhu added a comment - Just to confirm that: you agree that just sending the hostname from executors to driver. Right?

            That seems it might create a similar situation, but it was filed against a version of Spark that didn't even have the code that caused this particular regression. So my hunch is same symptom, different root cause, which may or may not have been fixed in newer Spark versions.

            vanzin Marcelo Masiero Vanzin added a comment - That seems it might create a similar situation, but it was filed against a version of Spark that didn't even have the code that caused this particular regression. So my hunch is same symptom, different root cause, which may or may not have been fixed in newer Spark versions.

            Yes, given the HDFS restrictions, that should be enough.

            vanzin Marcelo Masiero Vanzin added a comment - Yes, given the HDFS restrictions, that should be enough.
            zsxwing Shixiong Zhu added a comment -

            tleftwich Could help test https://github.com/apache/spark/pull/13741 in your environment, please?

            zsxwing Shixiong Zhu added a comment - tleftwich Could help test https://github.com/apache/spark/pull/13741 in your environment, please?
            zsxwing Shixiong Zhu added a comment -

            FYI, I reverted SPARK-15395 for branch-1.6.

            zsxwing Shixiong Zhu added a comment - FYI, I reverted SPARK-15395 for branch-1.6.

            zsxwing I'll run the tests here shortly and get back to you.

            tleftwich Trystan Leftwich added a comment - zsxwing I'll run the tests here shortly and get back to you.

            zsxwing I've tested your fix locally and it all is working as expected. Thanks.

            tleftwich Trystan Leftwich added a comment - zsxwing I've tested your fix locally and it all is working as expected. Thanks.
            zsxwing Shixiong Zhu added a comment -

            tleftwich Thanks! I'm merging it into 2.0 now!

            zsxwing Shixiong Zhu added a comment - tleftwich Thanks! I'm merging it into 2.0 now!

            People

              zsxwing Shixiong Zhu
              tleftwich Trystan Leftwich
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: