[SPARK-21171] Speculate task scheduling block dirve handle normal task when a job task number more than one hundred thousand

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 2.0.0
• Component/s: Block Manager, Spark Core

Environment

We have more than two hundred high-performance machine to handle more than 2T data by one query

• Assignee: Unassigned
• Reporter: wangminfeng
• Created: 22/Jun/17 05:45
• Updated: 17/May/20 18:21
• Resolved: 23/Jun/17 08:14

Description

If a job have more then one hundred thousand tasks and spark.speculation is true, when speculable tasks start, choosing a speculable will waste lots of time and block other tasks. We do a ad-hoc query for data analyse, we can't tolerate one job wasting time even it is a large job