[NIFI-5456] PutKinesisStream - Fails to work with AWS Private Link endpoint

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.0, 1.7.1
• Fix Version/s: 1.8.0
• Component/s: Extensions
• Labels: easyfix
• Environment: RedHat 6
• Assignee: Sivaprasanna Sethuraman
• Reporter: Ariel Godinez
• Created: 25/Jul/18 18:33
• Updated: 27/Feb/24 12:37
• Resolved: 05/Sep/18 15:43

Description

NiFi version: 1.6.0

PutKinesisStream fails to put due to invalid signing information when using an AWS Private Link as the endpoint override URL. The endpoint override URL pattern for private links is like below along with the error that NiFi outputs when we attempt to use this type of URL as the 'Endpoint Override URL' property value.

Endpoint Override URL: https://vpce-<AWS_GENERATED_ALPHA_NUMERIC>.kinesis.us-east-2.vpce.amazonaws.com

ERROR [Timer-Driven Process Thread-11] "o.a.n.p.a.k.stream.PutKinesisStream" PutKinesisStream[id=4c314e25-0164-1000-ffff-ffff9bd79c77] Failed to publish due to exception com.amazonaws.services.kinesis.model.AmazonKinesisException: Credential should be scoped to a valid region, not 'vpce'.  (Service: AmazonKinesis; Status Code: 400; Error Code: InvalidSignatureException; Request ID: 6330b83c-a64e-4acf-b892-a505621cf78e) flowfiles [StandardFlowFileRecord[uuid=ba299cec-7cbf-4750-a766-c348b5cd9c73,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1532469012962-1, container=content002, section=1], offset=2159750, length=534625],offset=0,name=900966573101260,size=534625]]

 

It looks like 'vpce' is being extracted from the url as the region name when it should be getting 'us-east-2'. We were able to get this processor to work correctly by explicitly passing in the region and service using 'setEndpoint(String endpoint, String serviceName, String regionId)' instead of 'setEndpoint(String endpoint)' in 'nifi/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java' line 289

Issue Links

• is related to: NIFI-8662 Failed to parse AWS region from VPCE endpoint URL in AbstractAWSProcessor
• is related to: NIFI-12846 AWS Assume Role Credentials with VPCE Endpoint URL cannot handle the Region
• links to: GitHub Pull Request #2968
• links to: GitHub Pull Request #2968