[SPARK-24218] Allow Configuration of DynamoDbEndpointUrl in KinesisReceiver

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.3.0
• Component/s: DStreams
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Cole Murray
• Shepherd: Cole Murray
• Created: 09/May/18 05:16
• Updated: 08/Oct/19 05:42
• Resolved: 08/Oct/19 05:42

Description

KinesisReceiver does not allow users to set DynamoDbEndpointUrl. When testing Kinesis locally, users are unable to utilize local checkpoints through a local Dynamodb.

 

An update to the constructor would allow other Apis, specifically, KinesisInputDStream.builder to add support for local DynamoDb endpointUrls.

 

Resources:

[https://github.com/apache/spark/blob/master/external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala|KinesisReceiver]

[https://github.com/apache/spark/blob/master/external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala#L162|Line To Add EndpointUrl to KinesisClientLibConfiguration]