Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-937

PutS3 Processor cannot upload files larger than 5 GB

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Duplicate
    • Affects Version/s: 0.3.0
    • Fix Version/s: 0.5.0
    • Component/s: Extensions
    • Labels:

      Description

      Tried to use the PutS3 processor to upload a large file. Got the error shown below.

      S3 doesn't allow you to upload > 5GB unless you use multi-part upload. It'd be nice if the processor handled this transparently, as the aws cli does.

      2015-09-08 14:20:38,991 ERROR [Timer-Driven Process Thread-6] o.a.nifi.processors.aws.s3.PutS3Object
      PutS3Object[id=14280a26-a34a-47f6-aa7e-df404c960634] Failed to put StandardFlowFileRecord[uuid=ba7e
      8332-4140-44f4-a71d-e5529f0ab6ad,claim=1441721637937-1157587,offset=0,name=redacted,size
      =5672981796] to Amazon S3 due to com.amazonaws.services.s3.model.AmazonS3Exception: Your proposed up
      load exceeds the maximum allowed size (Service: Amazon S3; Status Code: 400; Error Code: EntityTooLa
      rge; Request ID: 56B1AE1E6D8068C0), S3 Extended Request ID: lQImEwX+c7TUC3+Qr97kFult/95qinlRTFM/CF6v
      Z4b9DQwbwCn6auAc8epWlfHmHaU3AZCEvDc=: com.amazonaws.services.s3.model.AmazonS3Exception: Your propos
      ed upload exceeds the maximum allowed size (Service: Amazon S3; Status Code: 400; Error Code: Entity
      TooLarge; Request ID: 56B1AE1E6D8068C0), S3 Extended Request ID: lQImEwX+c7TUC3+Qr97kFult/95qinlRTFM
      /CF6vZ4b9DQwbwCn6auAc8epWlfHmHaU3AZCEvDc=

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jskora Joe Skora
                Reporter:
                johntitus John Titus
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: