[SPARK-20662] Block jobs that have greater than a configured number of tasks

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 1.6.0, 2.0.0
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Xuefu Zhang
• Created: 08/May/17 21:25
• Updated: 09/Jun/17 17:53
• Resolved: 09/Jun/17 17:53

Description

In a shared cluster, it's desirable for an admin to block large Spark jobs. While there might not be a single metrics defining the size of a job, the number of tasks is usually a good indicator. Thus, it would be useful for Spark scheduler to block a job whose number of tasks reaches a configured limit. By default, the limit could be just infinite, to retain the existing behavior.

MapReduce has mapreduce.job.max.map and mapreduce.job.max.reduce to be configured, which blocks a MR job at job submission time.

The proposed configuration is spark.job.max.tasks with a default value -1 (infinite).