[NIFI-8662] Failed to parse AWS region from VPCE endpoint URL in AbstractAWSProcessor

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.14.0
• Assignee: Peter Turcsanyi
• Reporter: Peter Turcsanyi
• Created: 04/Jun/21 18:37
• Updated: 27/Feb/24 12:38
• Resolved: 16/Jun/21 20:23

Description

The AWS client library cannot parse the region from custom endpoint URLs properly.
NIFI-5456 fixed this issue via passing the region configured on the processor to AmazonWebServiceClient.setEndpoint() directly (no parsing needed in the client library, neither in NiFi).
NIFI-5893 implemented the fix in another way: parsing the region from the endpoint URL on the NiFi side. It is not clear for me what special use case it wanted to solve but a regular VPCE endpoint does not work with it now.

Endpoint URL: https://vpce-*****************-********.sqs.us-west-2.vpce.amazonaws.com
Error:

2021-06-04 18:25:57,101 ERROR [Timer-Driven Process Thread-5] o.apache.nifi.processors.aws.sqs.PutSQS PutSQS[id=c4714170-c2cb-39e9-a36c-c43e4604f64a] Failed to send messages to Amazon SQS due to com.amazonaws.services.sqs.model.AmazonSQSException: Credential should be scoped to a valid region, not 'us-east-1'. (Service: AmazonSQS; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: 63ea72ad-a856-5eca-8c00-2b99da238d07)

It seems the "sqs" part does not match the regex used for parsing: ^(?:.[vpce-][a-z0-9-]\.)?([a-z0-9-]+)$.

However, the endpoint properly works with NIFI-5456 only.

To support both fixes, I will implement the following logic:

• use the parse method from NIFI-5893 first and use that region if the parse successful (no change here)
• if the parsing fails, then fall back to NIFI-5456 (that is using the configured region) instead of the hard coded "us-east-1"

Issue Links

• is related to: NIFI-12846 AWS Assume Role Credentials with VPCE Endpoint URL cannot handle the Region
• relates to: NIFI-5456 PutKinesisStream - Fails to work with AWS Private Link endpoint
• relates to: NIFI-5893 Wrong region id is being used for custom region in AbstractAWSProcessor
• links to: GitHub Pull Request #5140