Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22123

Add latest failure reason for task set blacklist

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.1.0, 2.2.0
    • Fix Version/s: 2.3.0
    • Component/s: Scheduler, Spark Core
    • Labels:
      None

      Description

      Till now , every job which aborted by completed blacklist just show log like below which has no more information:

      Aborting $taskSet because task $indexInTaskSet (partition $partition) cannot run anywhere due to node and executor blacklist. Blacklisting behavior cannot run anywhere due to node and executor blacklist.Blacklisting behavior can be configured via spark.blacklist.*."
      

      We could add most recent failure reason for taskset blacklist which can be showed on spark ui to let user know failure reason directly.
      An example after modifying:

      Aborting TaskSet 0.0 because task 0 (partition 0) cannot run anywhere due to node and executor blacklist.
       Most recent failure:
       Some(Lost task 0.1 in stage 0.0 (TID 3,xxx, executor 1): java.lang.Exception: Fake error!
       at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:73)
       at org.apache.spark.scheduler.Task.run(Task.scala:99)
       at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:305)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
       at java.lang.Thread.run(Thread.java:745)
       ). 
      
      Blacklisting behavior can be configured via spark.blacklist.*.
      

        Attachments

          Activity

            People

            • Assignee:
              cane zhoukang
              Reporter:
              cane zhoukang
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: