[SPARK-5959] Create a ReliableKinesisReceiver similar to the ReliableKafkaReceiver

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.1.0
• Component/s: DStreams
• Target Version/s: 1.5.0
• Assignee: Chris Fregly
• Reporter: Chris Fregly
• Created: 24/Feb/15 01:48
• Updated: 01/Aug/15 06:09
• Resolved: 30/Apr/15 08:03

Description

After each block is stored reliably in the WAL (after the store() call returns), ACK back to Kinesis.

There is still the issue of the ReliableKinesisReceiver dying before the ACK back to Kinesis, however no data will be lost. Duplicate data is still possible.

Notes:

• Make sure we're not overloading the checkpoint control plane which uses DynamoDB.
• May need to disable auto-checkpointing and remove the checkpoint interval.
• Maintain compatibility with existing KinesisReceiver-based code.

Issue Links

• duplicates: SPARK-9215 Implement WAL-free Kinesis receiver that give at-least once guarantee