In 0.5.0, I am seeing an error putting objects to Amazon S3 using the PutS3Object processor. The put operation itself succeeds, but a nasty-gram is displayed in the flow and logged as a side effect. The error text is "Error checking S3 Multipart Upload list for <my s3 bucket>", from line 698 of PutS3Object.java. This code was introduced in
NIFI-1107, expanding the PutS3Object processor with support for S3 multi-part uploads of larger files.
Logged in nifi-app.log:
The error-but-success behavior appears to happen because the exception is caught, logged, but not re-thrown. The approximate code path is as follows:
- PutS3Object::onTrigger method calls ageoffS3Uploads regardless of single-part/multi-part code path
- ageoffS3Uploads calls getS3AgeoffListAndAgeoffLocalState
- getS3AgeoffListAndAgeoffLocalState calls s3.listMultipartUploads
- After the exception, onTrigger proceeds with the single-part upload path
Listing multi-part S3 uploads requires an additional S3 permission, s3:ListBucketMultipartUploads. I would expect the PutS3Object processor to only require the s3:PutObject permission for single-part uploads.
I tested that this behavior did not occur in NiFi 0.4.2. I also tested that adding the s3:ListBucketMultipartUploads permission to my AWS credential circumvents the error messages from the S3PutObject processor.