Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26614

Speculation kill might cause job failure

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.0
    • Fix Version/s: 2.2.2, 2.3.1, 2.4.0
    • Component/s: Scheduler, Spark Core
    • Labels:
      None

      Description

      This issue is similar to SPARK-26612

      Some odd exceptions might be thrown in speculation kill, however, currently spark does not handle this case and will report failure to Driver. This exception will be counting towards MAX_TASK_FAILURES, and might result in the job failure.

       

      I think we can check state of task to tell if we report failure or killed.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                liupengcheng liupengcheng
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: