[FLINK-19716] Unable to use Assume Role with EFO record publisher

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.12.0
• Component/s: Connectors / Kinesis
• Labels: pull-request-available
• Assignee: Danny Cranmer
• Reporter: Danny Cranmer
• Created: 19/Oct/20 13:05
• Updated: 07/Dec/22 10:34
• Resolved: 20/Oct/20 05:21

Description

Reproduction Steps

• Setup an application to consume from a Kinesis Stream
• Use ASSUME_ROLE credential provider 

consumerConfig.setProperty(AWSConfigConstants.AWS_CREDENTIALS_PROVIDER, ASSUME_ROLE.name());
consumerConfig.setProperty(AWSConfigConstants.AWS_ROLE_ARN, "<role-arn>");
consumerConfig.setProperty(AWSConfigConstants.AWS_ROLE_SESSION_NAME, "test-efo");

 

Expected Result

• Consumer is able to authorise and consume from the stream

Actual Result

• The following error is thrown (full stack attached)
  • Caused by: org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.exception.SdkClientException: Unable to load an HTTP implementation from any provider in the chain. You must declare a dependency on an appropriate HTTP implementation or pass in an SdkHttpClient explicitly to the client builder.

Diagnosis

This issue occurs because Assume Role credential provider requires a Sync HTTP Client. The Apache HTTP Client is on the classpath but it is not detected due to the shading relocation. It is looking for:

• org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.SdkHttpService

However the service manifest is defining:

• software.amazon.awssdk.http.SdkHttpService

Solution

• Add a service manifest such that the shaded HTTP client is used
• Also needed to update the HTTP client/core version due to incompatibilities 

Testing

Tested using EFO and POLLING record consumer 

 

Issue Links

• links to: GitHub Pull Request #13689