Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-49001

Add a config to add ability to only enable speculative execution on non-data-writing tasks.

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • 3.3.2
    • None
    • Spark Core
    • None

    Description

      Speculative Execution is a very good tool to avoid straggler tasks, but if we use Speculative Execution with some customized BatchWrite, if there are bugs in user implemented BatchWrite, it will produce wrong results.

       

      So this ticket propose to add a config to add the ability to only enable speculative execution on non-data-writing tasks, which might be more safe.

      Attachments

        Activity

          People

            Unassigned Unassigned
            xumingming James Xu
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated: