[CAMEL-21031] camel-aws-sqs: support for polling more than 10 messages from AWS SQS

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.7.0
• Fix Version/s: 4.8.0
• Component/s: camel-aws2-sqs
• Estimated Complexity: Unknown
• Assignee: Otavio Rodolfo Piske
• Reporter: Otavio Rodolfo Piske
• Created: 30/Jul/24 08:26
• Updated: 30/Jul/24 08:27
• Resolved: 30/Jul/24 08:27

Description

Quoting Jakub from the PR:

??Hi,
I'm using camel-aws-sqs-source-kafka-connector, and recently I've found that I'm unable to poll all the messages from SQS queue in time.
At first I wanted to increase the maximum number of messages polled in a single request.
By default the client is configured to receive only 1 message and the maximum messages allowed by AWS is 10.
Here I had to add maxMessagesPerPoll parameter to the Kamelet in order to configure it for the connector.
I'm mentioning the Kamelet because the parameter there has maximum: 10 but this change renders that constraint obsolete.

Without the Kamelet there is a workaround to configure the queueNameOrArn with ?maxMessagesPerPoll=10& suffix, to sneak in the parameter to the endpoint.
But even with that and enabled greedy polling I was unable to keep up, and poll all messages in time.
Increasing the number of concurrent polling threads is an option here, but I'm interested in processing the messages in the correct order.

Looking at the Spring implementation, I've found that they are invoking receive requests with multiple threads and then merging the results together:
https://github.com/awspring/spring-cloud-aws/blob/a81b879d1a6b46f8a26794b98a601f7450d84907/spring-cloud-aws-sqs/src/main/java/io/awspring/cloud/sqs/listener/source/AbstractSqsMessageSource.java#L194-L198

I think a similar approach could be applied in Apache Camel, where the number of concurrent requests is calculated from ceil(maxMessagesPerPoll / 10D) formula.??