Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-16654

UI Should show blacklisted executors & nodes

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.2.0
    • Component/s: Scheduler, Web UI
    • Labels:
      None

      Description

      SPARK-8425 will add the ability to blacklist entire executors and nodes to deal w/ faulty hardware. However, without displaying it on the UI, it can be hard to realize which executor is bad, and why tasks aren't getting scheduled on certain executors.

      As a first step, we should just show nodes and executors that are blacklisted for the entire application (no need to show blacklisting for tasks & stages).

      This should also ensure that blacklisting events get into the event logs for the history server.

        Issue Links

          Activity

          Hide
          irashid Imran Rashid added a comment -
          Show
          irashid Imran Rashid added a comment - cc Thomas Graves
          Hide
          ajbozarth Alex Bozarth added a comment -

          Perhaps we can change the status column to "Blacklisted" or "Alive (Blacklisted)" instead on Alive or Dead? I'm not very familiar with how the blacklisting works but I would be willing to learn and add this once the other PR is merged.

          Show
          ajbozarth Alex Bozarth added a comment - Perhaps we can change the status column to "Blacklisted" or "Alive (Blacklisted)" instead on Alive or Dead? I'm not very familiar with how the blacklisting works but I would be willing to learn and add this once the other PR is merged.
          Hide
          jsoltren Jose Soltren added a comment -

          Alex, I'm trying to understand your July 20th comment. Are you saying that you would simply like to rename Alive/Dead, or that you might be interested in picking up some of the UI work?

          I was planning to start working on this somewhat soon. If you've made any steps here I'd like to have a look, and/or, you would be a welcome reviewer on any changes I send out.

          I'll attach a mock-up of the proposed UI changes at a later date.

          Show
          jsoltren Jose Soltren added a comment - Alex, I'm trying to understand your July 20th comment. Are you saying that you would simply like to rename Alive/Dead, or that you might be interested in picking up some of the UI work? I was planning to start working on this somewhat soon. If you've made any steps here I'd like to have a look, and/or, you would be a welcome reviewer on any changes I send out. I'll attach a mock-up of the proposed UI changes at a later date.
          Hide
          ajbozarth Alex Bozarth added a comment -

          I don't have time right now to tackle this so go right ahead. And the other part of my comment was a implementation suggestion. We currently have a "status" column that lists either Alive or Dead. I'm suggesting that when shown, Blacklisted nodes are listed as Blacklisted or Alive (Blacklisted) in the status column, this would make the ui change for this very minimal to the user even though it'll be a good chunk of code to make it work behind the scenes.

          Show
          ajbozarth Alex Bozarth added a comment - I don't have time right now to tackle this so go right ahead. And the other part of my comment was a implementation suggestion. We currently have a "status" column that lists either Alive or Dead. I'm suggesting that when shown, Blacklisted nodes are listed as Blacklisted or Alive (Blacklisted) in the status column, this would make the ui change for this very minimal to the user even though it'll be a good chunk of code to make it work behind the scenes.
          Hide
          jsoltren Jose Soltren added a comment -

          SPARK-8425 is resolved so I'll be working to get this checked in fairly soon now.

          Show
          jsoltren Jose Soltren added a comment - SPARK-8425 is resolved so I'll be working to get this checked in fairly soon now.
          Hide
          apachespark Apache Spark added a comment -

          User 'jsoltren' has created a pull request for this issue:
          https://github.com/apache/spark/pull/16346

          Show
          apachespark Apache Spark added a comment - User 'jsoltren' has created a pull request for this issue: https://github.com/apache/spark/pull/16346
          Hide
          irashid Imran Rashid added a comment -

          Issue resolved by pull request 16346
          https://github.com/apache/spark/pull/16346

          Show
          irashid Imran Rashid added a comment - Issue resolved by pull request 16346 https://github.com/apache/spark/pull/16346

            People

            • Assignee:
              jsoltren Jose Soltren
              Reporter:
              irashid Imran Rashid
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development