Major Description
AWS2S3 Producer comes in two flavours: AWS2S3Producer and AWS2S3StreamUploadProducer.
- AWS2S3Producer supports S3 multipart upload: exchange data will be chunked into parts, parts will be sent separately to S3 and will be aggregated into one file.
- AWS2S3StreamUploadProducer supports message streaming: incoming messages are aggregated into a buffer that will be sent as one file to S3, as soon as some condition is true (number of messages / size /timeout).
Unfortunately, AWS2S3StreamUploadProducer, although it is able to break a huge flow of messages into parts (which is great for some use cases), is not able to manage S3 multipart upload. So if you need to aggregate lots of messages into one big file on S3 side, that means the producer will have to send that message batch in one request (actually, a single part multipart request if you look at implementation). If the volume is huge, that will blow up jvm heap.
So we propose to add multi-part support to AWS2S3StreamUploadProducer.
Attachments
Issue Links
- links to