[FLINK-31108] Use StreamARN for API calls in Kinesis Connector

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Done
• Affects Version/s: 1.15.3, aws-connector-4.1.0
• Fix Version/s: aws-connector-4.2.0
• Component/s: Connectors / Kinesis
• Labels: pull-request-available
• Assignee: Hong Liang Teoh
• Reporter: Hong Liang Teoh
• Created: 16/Feb/23 17:05
• Updated: 02/May/23 10:07
• Resolved: 02/May/23 10:07

Description

Currently, the FlinkKinesisConsumer (Polling + EFO) + FlinkKinesisProducer uses the stream name during API calls

We want to change this to the StreamARN. There are two reasons for this:

• This allows lower latency calls to the Kinesis endpoint for GetRecords API
• Paves the way for allowing user target cross-account streams without assume role (i.e. IAM role in account A but target stream in account B)

 

The APIs that are currently called:

• ListShards
• GetShardIterator
• GetRecords
• DescribeStream
• DescribeStreamSummary
• DescribeStreamConsumer (already uses StreamARN)
• RegisterStreamConsumer (already uses StreamARN)

 

Issue Links

• links to: GitHub Pull Request #65