[BEAM-3638] S3 occasionally fails to delete

• Type: Bug
• Status: Resolved
• Priority: P1
• Resolution: Invalid
• Affects Version/s: 2.3.0
• Fix Version/s: Not applicable
• Component/s: io-java-aws
• Assignee: Jacob Marble
• Reporter: Jacob Marble
• Created: 07/Feb/18 17:03
• Updated: 24/Jul/20 20:20
• Resolved: 07/Feb/18 17:20

Description

When AmazonS3Client.deleteObjects() is called, the S3 service occasionally returns a retryable error, and also deletes the object. Beam retries the request, and the S3 service correctly returns a "404 Not Found" error. The job fails needlessly.

I propose to fix this by logging and allowing "404 Not Found" errors; if the object doesn't exist then the current state is also the desired state. Should have a PR later today.

The first error looks like this:

java.io.IOException: com.amazonaws.services.s3.model.MultiObjectDeleteException: One or more objects could not be deleted (Service: null; Status Code: 200; Error Code: null; Request ID: [redacted]; S3 Extended Request ID: [redacted]), S3 Extended Request ID: [redacted]

Subsequent errors:

java.io.IOException: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: [redacted]; S3 Extended Request ID: [redacted]), S3 Extended Request ID: [redacted]