[CAMEL-13433] S3: Exchange body stream is loaded into memory to calculate content length which is already set via headers

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.22.1, 2.23.2, 3.0.0-M1
• Fix Version/s: 2.24.0, 3.0.0-M3, 2.23.3, 3.0.0
• Component/s: camel-aws
• Estimated Complexity: Novice
• Regression: Regression
• Assignee: Andrea Cosentino
• Reporter: MykhailoVlakh
• Created: 18/Apr/19 10:20
• Updated: 23/Apr/19 09:17
• Resolved: 23/Apr/19 09:17

Description

My code uploads large files to S3 storage (500 MB and bigger). I create an exchange with properly set meta information headers including the CamelAwsS3ContentLength, as a body I provide a stream. Everything worked well in Camel 2.14.3 but when I upgraded to Camel ti version 2.22.1 I suddenly began getting OOM issues from time to time. I did some debugging and found that the issue is in S3Producer that loads content of the body stream into memory.

public void processSingleOp(final Exchange exchange) throws Exception {

ObjectMetadata objectMetadata = determineMetadata(exchange);  <== here we already set length

File filePayload = null;
InputStream is = null;
ByteArrayOutputStream baos = null;
Object obj = exchange.getIn().getMandatoryBody();
PutObjectRequest putObjectRequest = null;
// Need to check if the message body is WrappedFile
if (obj instanceof WrappedFile) {
   obj = ((WrappedFile<?>)obj).getFile();
}
if (obj instanceof File) {
   filePayload = (File)obj;
   is = new FileInputStream(filePayload);
} else {
   is = exchange.getIn().getMandatoryBody(InputStream.class);
   baos = determineLengthInputStream(is);        <====== the issue is here
   objectMetadata.setContentLength(baos.size());
   is = new ByteArrayInputStream(baos.toByteArray());
}

...

This is a defect since, as I mentioned, I already provide content length via headers and in the first line of the method populates that information into object metadata.

I think there should be some if check that skips content length size calculation if it is already known.