[SPARK-27950] Additional DynamoDB and CloudWatch config

• Type: Improvement
• Status: In Progress
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: Structured Streaming
• Labels: pull-request-available
• Flags: Patch
• Assignee: Unassigned
• Reporter: Eric S Meisel
• Created: 05/Jun/19 02:38
• Updated: 04/Apr/24 17:31

Description

I was researching getting Spark’s Kinesis integration running locally against localstack. We found this issue, and it creates a complication: localstack/localstack#677

Effectively, we need to be able to redirect calls for Kinesis, DynamoDB and Cloudwatch in order for the KCL to properly use the localstack infrastructure. We have successfully done this with the KCL (both 1.x and 2.x), but with Spark’s integration we are unable to configure DynamoDB and Cloudwatch’s endpoints:

https://github.com/apache/spark/blob/master/external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala#L162

Issue Links

• links to: [Github] Pull Request #24801 (etspaceman)
• links to: GitHub Pull Request #24801
• links to: GitHub Pull Request #45619