[SPARK-4062] Improve KafkaReceiver to prevent data loss

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: DStreams
• Target Version/s: 1.2.0
• Assignee: Saisai Shao
• Reporter: Saisai Shao
• Created: 23/Oct/14 13:31
• Updated: 14/Nov/14 22:35
• Resolved: 14/Nov/14 22:35

Description

Current KafkaReceiver has data loss and data re-consuming problem. Here we propose a ReliableKafkaReceiver to improving its reliability and fault tolerance with the power of Spark Streaming's WAL mechanism.

This is a follow up work of SPARK-3129. Design doc is posted, any comments would be greatly appreciated.

Issue Links

• relates to: SPARK-3129 Prevent data loss in Spark Streaming on driver failure using Write Ahead Logs
• links to: [Github] Pull Request #2991 (jerryshao)