Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-1542

PutS3Object Processor Logs Errors Without ListBucketMultipartUploads Permission

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.5.0
    • Fix Version/s: 0.5.1
    • Component/s: Extensions
    • Labels:
      None
    • Environment:
      Linux, Chrome

      Description

      In 0.5.0, I am seeing an error putting objects to Amazon S3 using the PutS3Object processor. The put operation itself succeeds, but a nasty-gram is displayed in the flow and logged as a side effect. The error text is "Error checking S3 Multipart Upload list for <my s3 bucket>", from line 698 of PutS3Object.java. This code was introduced in NIFI-1107, expanding the PutS3Object processor with support for S3 multi-part uploads of larger files.

      Logged in nifi-app.log:

      2016-02-19 20:56:24,312 ERROR [Timer-Driven Process Thread-3] o.a.nifi.processors.aws.s3.PutS3Object PutS
      3Object[id=47df7533-91b2-4635-b865-45323f73f6c3] Error checking S3 Multipart Upload list for batchiq-scra
      tch: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 7483E76EA
      52E33C4)
      
      2016-02-19 20:56:24,533 INFO [Timer-Driven Process Thread-3] o.a.nifi.processors.aws.s3.PutS3Object PutS3
      Object[id=47df7533-91b2-4635-b865-45323f73f6c3] Successfully put StandardFlowFileRecord[uuid=a4cc19f5-cd2
      4-4be7-8157-21f2609fa8cd,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1455912275117-1, container=default, section=1], offset=3072, length=1024],offset=0,name=273670477119454,size=1024] to Amazon S3 in 327 milliseconds
      

      The error-but-success behavior appears to happen because the exception is caught, logged, but not re-thrown. The approximate code path is as follows:

      1. PutS3Object::onTrigger method calls ageoffS3Uploads regardless of single-part/multi-part code path
      2. ageoffS3Uploads calls getS3AgeoffListAndAgeoffLocalState
      3. getS3AgeoffListAndAgeoffLocalState calls s3.listMultipartUploads
      4. After the exception, onTrigger proceeds with the single-part upload path

      Troubleshooting Notes
      Listing multi-part S3 uploads requires an additional S3 permission, s3:ListBucketMultipartUploads. I would expect the PutS3Object processor to only require the s3:PutObject permission for single-part uploads.

      I tested that this behavior did not occur in NiFi 0.4.2. I also tested that adding the s3:ListBucketMultipartUploads permission to my AWS credential circumvents the error messages from the S3PutObject processor.

        Attachments

          Activity

            People

            • Assignee:
              jskora Joe Skora
              Reporter:
              jameswing James Wing
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: