[NIFI-6724] PutSQS does not check for individual status in batch message

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.9.2
• Fix Version/s: 1.11.0
• Component/s: Extensions
• Labels: Bug PutSQS SQS
• Environment: Docker running in AWS
• Assignee: Pierre Villard
• Reporter: Martin
• Created: 27/Sep/19 13:13
• Updated: 22/Nov/19 16:32
• Resolved: 22/Nov/19 16:32

Description

PutSQS does not check for individual status in batch message. This could lead to PutSQS reporting success and dropping files even when individual files actually failed in SQS. 

As specified in the AWSSimpleQueueService API Reference for SendMessageBatch:

The result of sending each message is reported individually in the response. Because the batch request can result in a combination of successful and unsuccessful actions, you should check for batch errors even when the call returns an HTTP status code of 200.

 

We learned this as we did not Base64-encode compressed files (gzip) before sending them to PutSQS. All the compressed files failed to queue in SQS, but SQS acknowledged the batch message and reported status 200 (success). Thus PutSQS also reported success and dropped the files, when they actually failed. 

I suggest a simple fix by implementing SendMessageBatchResult to check status on individual files by implementing something similar to this

SendMessageBatchResult sendMessageBatchResult = sqsClient.sendMessageBatch(request);
                if (!sendMessageBatchResult.getFailed().isEmpty()) {
                    // Error
                }

Issue Links

• links to: GitHub Pull Request #3897