Uploaded image for project: 'Camel'
  1. Camel
  2. CAMEL-13433

S3: Exchange body stream is loaded into memory to calculate content length which is already set via headers

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.22.1, 2.23.2, 3.0.0-M1
    • Fix Version/s: 2.24.0, 3.0.0-M3, 2.23.3, 3.0.0
    • Component/s: camel-aws
    • Labels:
      None
    • Estimated Complexity:
      Novice
    • Regression:
      Regression

      Description

      My code uploads large files to S3 storage (500 MB and bigger). I create an exchange with properly set meta information headers including the CamelAwsS3ContentLength, as a body I provide a stream. Everything worked well in Camel 2.14.3 but when I upgraded to Camel ti version 2.22.1 I suddenly began getting OOM issues from time to time. I did some debugging and found that the issue is in S3Producer that loads content of the body stream into memory.

      public void processSingleOp(final Exchange exchange) throws Exception {
      
      ObjectMetadata objectMetadata = determineMetadata(exchange);  <== here we already set length
      
      File filePayload = null;
      InputStream is = null;
      ByteArrayOutputStream baos = null;
      Object obj = exchange.getIn().getMandatoryBody();
      PutObjectRequest putObjectRequest = null;
      // Need to check if the message body is WrappedFile
      if (obj instanceof WrappedFile) {
         obj = ((WrappedFile<?>)obj).getFile();
      }
      if (obj instanceof File) {
         filePayload = (File)obj;
         is = new FileInputStream(filePayload);
      } else {
         is = exchange.getIn().getMandatoryBody(InputStream.class);
         baos = determineLengthInputStream(is);        <====== the issue is here
         objectMetadata.setContentLength(baos.size());
         is = new ByteArrayInputStream(baos.toByteArray());
      }
      
      ...
      

      This is a defect since, as I mentioned, I already provide content length via headers and in the first line of the method populates that information into object metadata.

      I think there should be some if check that skips content length size calculation if it is already known. 

       

        Attachments

          Activity

            People

            • Assignee:
              acosentino Andrea Cosentino
              Reporter:
              mvlakh MykhailoVlakh
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: