[SPARK-10694] Prevent Data Loss in Spark Streaming when used with OFF_HEAP ExternalBlockStore (Tachyon)

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 1.5.0
• Component/s: Block Manager, DStreams, Spark Core
• Assignee: Unassigned
• Reporter: Dibyendu Bhattacharya
• Created: 18/Sep/15 12:08
• Updated: 17/May/20 18:20
• Resolved: 01/Mar/16 19:18

Description

If Streaming application stores the blocks OFF_HEAP, it may not need any WAL like feature to recover from Driver failure. As long as the writing of blocks to Tachyon from Streaming receiver is durable, it should be recoverable from Tachyon directly on Driver failure.
This can solve the issue of expensive WAL write and duplicating the blocks both in MEMORY and also WAL and also guarantee end to end No-Data-Loss channel using OFF_HEAP store.

Issue Links

• links to: [Github] Pull Request #8817 (dibbhatt)