[SPARK-10692] Failed batches are never reported through the StreamingListener interface

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.3.1, 1.4.1, 1.5.0
• Fix Version/s: 1.5.1, 1.6.0
• Component/s: DStreams
• Target Version/s: 1.5.1, 1.6.0
• Assignee: Shixiong Zhu
• Reporter: Tathagata Das
• Created: 18/Sep/15 11:11
• Updated: 24/Sep/15 02:55
• Resolved: 24/Sep/15 02:55

Description

If an output operation fails, then corresponding batch is never marked as completed, as the data structure are not updated properly.

https://github.com/apache/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobScheduler.scala#L183

Issue Links

• links to: [Github] Pull Request #8818 (zsxwing)
• links to: [Github] Pull Request #8892 (tdas)