[SPARK-9215] Implement WAL-free Kinesis receiver that give at-least once guarantee

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.1
• Fix Version/s: 1.5.0
• Component/s: DStreams
• Target Version/s: 1.5.0
• Assignee: Tathagata Das
• Reporter: Tathagata Das
• Created: 21/Jul/15 10:01
• Updated: 16/May/17 15:33
• Resolved: 05/Aug/15 07:21

Description

Currently, the KinesisReceiver can loose some data in the case of certain failures (receiver and driver failures). Using the write ahead logs can mitigate some of the problem, but it is not ideal because WALs dont work with S3 (eventually consistency, etc.) which is the most likely file system to be used in the EC2 environment. Hence, we have to take a different approach to improving reliability for Kinesis.

Detailed design doc - https://docs.google.com/document/d/1k0dl270EnK7uExrsCE7jYw7PYx0YC935uBcxn3p0f58/edit?usp=sharing

Issue Links

• is duplicated by: SPARK-5959 Create a ReliableKinesisReceiver similar to the ReliableKafkaReceiver
• is related to: SPARK-6599 Improve usability and reliability of Kinesis stream

Sub-Tasks

• Allow received block metadata to be saved to WAL and recovered on driver failure: Resolved
• Add method to BlockGenerator to add multiple records to BlockGenerator with single callback: Resolved
• Define KinesisBackedBlockRDDs: Resolved
• Update Kinesis Receiver to record sequence numbers: Resolved