[SPARK-6415] Spark Streaming fail-fast: Stop scheduling jobs when a batch fails, and kills the app

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Component/s: DStreams
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Hari Shreedharan
• Created: 19/Mar/15 18:02
• Updated: 21/May/19 05:37
• Resolved: 21/May/19 05:37

Description

Of course, this would have to be done as a configurable param, but such a fail-fast is useful else it is painful to figure out what is happening when there are cascading failures. In some cases, the SparkContext shuts down and streaming keeps scheduling jobs

Issue Links

• relates to: SPARK-4545 If first Spark Streaming batch fails, it waits 10x batch duration before stopping