[NIFI-937] PutS3 Processor cannot upload files larger than 5 GB

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 0.3.0
• Fix Version/s: 0.5.0
• Component/s: Extensions
• Labels: beginner
• Assignee: Joe Skora
• Reporter: John Titus
• Created: 08/Sep/15 14:34
• Updated: 04/Jan/16 19:46
• Resolved: 04/Jan/16 19:45

Description

Tried to use the PutS3 processor to upload a large file. Got the error shown below.

S3 doesn't allow you to upload > 5GB unless you use multi-part upload. It'd be nice if the processor handled this transparently, as the aws cli does.

2015-09-08 14:20:38,991 ERROR [Timer-Driven Process Thread-6] o.a.nifi.processors.aws.s3.PutS3Object
PutS3Object[id=14280a26-a34a-47f6-aa7e-df404c960634] Failed to put StandardFlowFileRecord[uuid=ba7e
8332-4140-44f4-a71d-e5529f0ab6ad,claim=1441721637937-1157587,offset=0,name=redacted,size
=5672981796] to Amazon S3 due to com.amazonaws.services.s3.model.AmazonS3Exception: Your proposed up
load exceeds the maximum allowed size (Service: Amazon S3; Status Code: 400; Error Code: EntityTooLa
rge; Request ID: 56B1AE1E6D8068C0), S3 Extended Request ID: lQImEwX+c7TUC3+Qr97kFult/95qinlRTFM/CF6v
Z4b9DQwbwCn6auAc8epWlfHmHaU3AZCEvDc=: com.amazonaws.services.s3.model.AmazonS3Exception: Your propos
ed upload exceeds the maximum allowed size (Service: Amazon S3; Status Code: 400; Error Code: Entity
TooLarge; Request ID: 56B1AE1E6D8068C0), S3 Extended Request ID: lQImEwX+c7TUC3+Qr97kFult/95qinlRTFM
/CF6vZ4b9DQwbwCn6auAc8epWlfHmHaU3AZCEvDc=

Issue Links

• duplicates: NIFI-1107 Add multipart upload support to PutS3Object