Uploaded image for project: 'Camel'
  1. Camel
  2. CAMEL-13433

S3: Exchange body stream is loaded into memory to calculate content length which is already set via headers

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.22.1, 2.23.2, 3.0.0-M1
    • 2.24.0, 3.0.0-M3, 2.23.3, 3.0.0
    • camel-aws
    • None
    • Novice
    • Regression

    Description

      My code uploads large files to S3 storage (500 MB and bigger). I create an exchange with properly set meta information headers including the CamelAwsS3ContentLength, as a body I provide a stream. Everything worked well in Camel 2.14.3 but when I upgraded to Camel ti version 2.22.1 I suddenly began getting OOM issues from time to time. I did some debugging and found that the issue is in S3Producer that loads content of the body stream into memory.

      public void processSingleOp(final Exchange exchange) throws Exception {
      
      ObjectMetadata objectMetadata = determineMetadata(exchange);  <== here we already set length
      
      File filePayload = null;
      InputStream is = null;
      ByteArrayOutputStream baos = null;
      Object obj = exchange.getIn().getMandatoryBody();
      PutObjectRequest putObjectRequest = null;
      // Need to check if the message body is WrappedFile
      if (obj instanceof WrappedFile) {
         obj = ((WrappedFile<?>)obj).getFile();
      }
      if (obj instanceof File) {
         filePayload = (File)obj;
         is = new FileInputStream(filePayload);
      } else {
         is = exchange.getIn().getMandatoryBody(InputStream.class);
         baos = determineLengthInputStream(is);        <====== the issue is here
         objectMetadata.setContentLength(baos.size());
         is = new ByteArrayInputStream(baos.toByteArray());
      }
      
      ...
      

      This is a defect since, as I mentioned, I already provide content length via headers and in the first line of the method populates that information into object metadata.

      I think there should be some if check that skips content length size calculation if it is already known. 

       

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            acosentino Andrea Cosentino
            mvlakh MykhailoVlakh
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment