[SPARK-10969] Spark Streaming Kinesis: Allow specifying separate credentials for Kinesis and DynamoDB

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 1.5.1
• Component/s: DStreams
• Assignee: Unassigned
• Reporter: Christoph Pirkl
• Created: 07/Oct/15 13:07
• Updated: 27/Nov/17 12:53
• Resolved: 27/Nov/17 12:53

Description

KinesisUtils.createStream() allows specifying only one set of AWS credentials that will be used by Amazon KCL for accessing Kinesis, DynamoDB and CloudWatch.

Motivation

In a scenario where one needs to read from a Kinesis Stream owned by a different AWS account the user usually has minimal rights (i.e. only read from the stream). In this case creating the DynamoDB table in KCL will fail.

Proposal

My proposed solution would be to allow specifying multiple credentials in KinesisUtils.createStream() for Kinesis, DynamoDB and CloudWatch. The additional credentials could then be passed to the constructor of KinesisClientLibConfiguration or method KinesisClientLibConfiguration.withDynamoDBClientConfig().

Issue Links

• duplicates: SPARK-19911 Add builder interface for Kinesis DStreams
• links to: [Github] Pull Request #11215 (kaklakariada)