[SPARK-5238] Improve the robustness of Spark Streaming WAL mechanism

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Done
• Affects Version/s: 1.2.0
• Component/s: DStreams
• Assignee: Unassigned
• Reporter: Saisai Shao
• Created: 14/Jan/15 05:52
• Updated: 19/Jan/16 14:08
• Resolved: 19/Jan/16 14:08

Description

Several issues identified in Spark Streaming's WAL mechanism, this is a cap of all the related issues.

Sub-Tasks

• Error replay of WAL when recovered from driver failue: Resolved
• write ahead logs from streaming receiver are not purged because cleanupOldBlocks in WriteAheadLogBasedBlockHandler is never called: Resolved