[SPARK-6599] Improve usability and reliability of Kinesis stream

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: DStreams
• Target Version/s: 1.5.0
• Assignee: Tathagata Das
• Reporter: Tathagata Das
• Created: 30/Mar/15 00:16
• Updated: 05/Aug/15 07:27
• Resolved: 05/Aug/15 07:27

Description

Usability improvements:
API improvements, AWS SDK upgrades, etc.

Reliability improvements:
Currently, the KinesisReceiver can loose some data in the case of certain failures (receiver and driver failures). Using the write ahead logs can mitigate some of the problem, but it is not ideal because WALs dont work with S3 (eventually consistency, etc.) which is the most likely file system to be used in the EC2 environment. Hence, we have to take a different approach to improving reliability for Kinesis. See https://issues.apache.org/jira/browse/SPARK-9215 for more details.

Issue Links

• relates to: SPARK-3638 Commons HTTP client dependency conflict in extras/kinesis-asl module
• relates to: SPARK-7679 Update AWS SDK and KCL versions to 1.2.1
• relates to: SPARK-5960 Allow AWS credentials to be passed to KinesisUtils.createStream()
• relates to: SPARK-6514 For Kinesis Streaming, use the same region for DynamoDB (KCL checkpoints) as the Kinesis stream itself
• relates to: SPARK-6656 Allow the application name to be passed in versus pulling from SparkContext.getAppName()
• relates to: SPARK-9215 Implement WAL-free Kinesis receiver that give at-least once guarantee
• relates to: SPARK-6654 Update Kinesis Streaming impls (both KCL-based and Direct) to use latest aws-java-sdk and kinesis-client-library
• relates to: SPARK-9030 Add Kinesis.createStream unit tests that actual sends data
• relates to: SPARK-4184 Improve Spark Streaming documentation to address commonly-asked questions