Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-20898

spark.blacklist.killBlacklistedExecutors doesn't work in YARN

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.3.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      I was trying out the new spark.blacklist.killBlacklistedExecutors on YARN but it doesn't appear to work. Everytime I get:

      17/05/26 16:28:12 WARN BlacklistTracker: Not attempting to kill blacklisted executor id 4 since allocation client is not defined

      Even though dynamic allocation is on. Taking a quick look, I think the way it creates the blacklisttracker and passes the allocation client is wrong. The scheduler backend is
      not set yet so it never passes the allocation client to the blacklisttracker correctly. Thus it will never kill.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jerryshao Saisai Shao
                Reporter:
                tgraves Thomas Graves
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: