Uploaded image for project: 'Camel'
  1. Camel
  2. CAMEL-15526

Camel-AWS2-S3: Consume Gzip file from S3 not working.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.4.2
    • Fix Version/s: 3.6.0, 3.4.5
    • Component/s: None
    • Labels:
      None
    • Environment:

      camel Aws2-s3

       

    • Estimated Complexity:
      Unknown

      Description

      Hi,

      When consuming Gzip file from S3 using AWS2-S3 component, file not downlaoding properly;

      Same code working for AWS-S3, after upgrading to AWS2-S3, this code not working.

       

      Task:
      1. Download "myfile.csv.gzip" file from S3.

      2. GUnzip file and send to processor.
      3. Processor will send file to http end point using multipart upload.

      Here is the code used.

       

      {{}}

      from("aws2-s3://mybucket?amazonS3Client=#amazonS3Client&deleteAfterRead=false&delay=5000&synchronous=true&includeBody=true&autocloseBody=false&prefix=myfile.csv.gz")

      {{ .unmarshal().gzipDeflater()}}

      {{ .process(@Override public void process(Exchange exchange) throws Exception {}}

      {{ MultipartEntityBuilder multipartEntityBuilder = MultipartEntityBuilder.create();
      String filename = exchange.getIn().getHeader(AWS2S3Constants.KEY, String.class);
      String body = exchange.getIn().getBody(String.class);
      ContentBody cd = new InputStreamBody(new ByteArrayInputStream(body.getBytes()), ContentType.MULTIPART_FORM_DATA, "temp.csv");
      multipartEntityBuilder.addPart("file", cd);
      System.out.println(cd);
      exchange.getOut().setBody(multipartEntityBuilder.build())
      .to("direct:httproute");}}

      {{}}

      Stack Trace:

      Stack Trace

      [main] WARN software.amazon.awssdk.profiles.internal.ProfileFileReader - Ignoring profile 'plugins' on line 6 because it did not start with 'profile ' and it was not 'default'.
      [main] INFO org.apache.camel.support.LRUCacheFactory - Detected and using LRUCacheFactory: camel-caffeine-lrucache
      [main] INFO org.apache.camel.component.http.HttpComponent - Created ClientConnectionManager org.apache.http.impl.conn.PoolingHttpClientConnectionManager@1b73be9f
      [main] INFO org.apache.camel.component.http.HttpComponent - Created ClientConnectionManager org.apache.http.impl.conn.PoolingHttpClientConnectionManager@35a9782c
      [main] INFO org.apache.camel.impl.engine.AbstractCamelContext - Apache Camel 3.4.2 (camel-1) is starting
      [main] INFO org.apache.camel.impl.engine.AbstractCamelContext - StreamCaching is not in use. If using streams then its recommended to enable stream caching. See more details at http://camel.apache.org/stream-caching.html
      [main] INFO org.apache.camel.impl.engine.AbstractCamelContext - Using HealthCheck: camel-health
      [main] INFO org.apache.camel.impl.engine.InternalRouteStartupManager - Route: route1 started and consuming from: aws2-s3://mybucket
      [main] INFO org.apache.camel.impl.engine.InternalRouteStartupManager - Route: route2 started and consuming from: direct://httproute
      [main] INFO org.apache.camel.impl.engine.AbstractCamelContext - Total 3 routes, of which 3 are started
      [main] INFO org.apache.camel.impl.engine.AbstractCamelContext - Apache Camel 3.4.2 (camel-1) started in 1.496 seconds
      [Camel (camel-1) thread #1 - aws2-s3://mybucket] INFO consuming - Consumer Fired!
      [Camel (camel-1) thread #1 - aws2-s3://mybucket] INFO route1 - Replay Message Sent to file:s3out input_2020-07-01-07-20-31-796_0.csv.gz
      [Camel (camel-1) thread #1 - aws2-s3://mybucket] ERROR org.apache.camel.processor.errorhandler.DefaultErrorHandler - Failed delivery for (MessageId: ID-DGHANTA-Mac-1599955541230-0-1 on ExchangeId: ID-DGHANTA-Mac-1599955541230-0-1). Exhausted after delivery attempt: 1 caught: java.util.zip.ZipException: Not in GZIP format

      Message History (complete message history is disabled)
      ---------------------------------------------------------------------------------------------------------------------------------------
      RouteId ProcessorId Processor Elapsed (ms)
      [route1 ] [route1 ] [from[aws2-s3://mybucket?amazonS3Client=%23] [ 250]
      ...
      [route1 ] [unmarshal1 ] [unmarshal[org.apache.camel.model.dataformat.GzipDataFormat@d41f816] ] [ 0]

      Stacktrace
      ---------------------------------------------------------------------------------------------------------------------------------------

      java.util.zip.ZipException: Not in GZIP format
      at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:165)
      at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:79)
      at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:91)
      at org.apache.camel.dataformat.deflater.GzipDeflaterDataFormat.unmarshal(GzipDeflaterDataFormat.java:62)
      at org.apache.camel.support.processor.UnmarshalProcessor.process(UnmarshalProcessor.java:64)
      at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$SimpleTask.run(RedeliveryErrorHandler.java:395)
      at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:148)
      at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:60)
      at org.apache.camel.processor.Pipeline.process(Pipeline.java:147)
      at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:286)
      at org.apache.camel.component.aws2.s3.AWS2S3Consumer.processBatch(AWS2S3Consumer.java:200)
      at org.apache.camel.component.aws2.s3.AWS2S3Consumer.poll(AWS2S3Consumer.java:113)
      at org.apache.camel.support.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:187)
      at org.apache.camel.support.ScheduledPollConsumer.run(ScheduledPollConsumer.java:106)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)

      Regards,

      Dinesh

      {{}}

        Attachments

          Activity

            People

            • Assignee:
              acosentino Andrea Cosentino
              Reporter:
              edingha Dinesh
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: