[SPARK-29424] Prevent Spark to committing stage of too much Task

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 2.4.0, 3.0.0
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: angerszhu
• Created: 10/Oct/19 09:40
• Updated: 26/Oct/19 21:00
• Resolved: 26/Oct/19 21:00

Description

Our user always submit bad SQL in query platform, Such as :

1. write wrong join condition but submit that sql
2. write wrong where condition
3. etc..

This case will make Spark scheduler to submit a lot of task. It will cause spark run very slow and impact other user(spark thrift server) even run out of memory because of too many object generated by a big num of tasks.

So I add a constraint when submit tasks and abort stage early when TaskSet size num is bigger then set limit . I wonder if the community will accept this way.
cc srowen dongjoon yumwang