Details
Description
Hi,
When consuming Gzip file from S3 using AWS2-S3 component, file not downlaoding properly;
Same code working for AWS-S3, after upgrading to AWS2-S3, this code not working.
Task:
1. Download "myfile.csv.gzip" file from S3.
2. GUnzip file and send to processor.
3. Processor will send file to http end point using multipart upload.
Here is the code used.
{{}}
from("aws2-s3://mybucket?amazonS3Client=#amazonS3Client&deleteAfterRead=false&delay=5000&synchronous=true&includeBody=true&autocloseBody=false&prefix=myfile.csv.gz")
{{ .unmarshal().gzipDeflater()}}
{{ .process(@Override public void process(Exchange exchange) throws Exception {}}
{{ MultipartEntityBuilder multipartEntityBuilder = MultipartEntityBuilder.create();
String filename = exchange.getIn().getHeader(AWS2S3Constants.KEY, String.class);
String body = exchange.getIn().getBody(String.class);
ContentBody cd = new InputStreamBody(new ByteArrayInputStream(body.getBytes()), ContentType.MULTIPART_FORM_DATA, "temp.csv");
multipartEntityBuilder.addPart("file", cd);
System.out.println(cd);
exchange.getOut().setBody(multipartEntityBuilder.build())
.to("direct:httproute");}}
{{}}
Stack Trace:
[main] WARN software.amazon.awssdk.profiles.internal.ProfileFileReader - Ignoring profile 'plugins' on line 6 because it did not start with 'profile ' and it was not 'default'.
[main] INFO org.apache.camel.support.LRUCacheFactory - Detected and using LRUCacheFactory: camel-caffeine-lrucache
[main] INFO org.apache.camel.component.http.HttpComponent - Created ClientConnectionManager org.apache.http.impl.conn.PoolingHttpClientConnectionManager@1b73be9f
[main] INFO org.apache.camel.component.http.HttpComponent - Created ClientConnectionManager org.apache.http.impl.conn.PoolingHttpClientConnectionManager@35a9782c
[main] INFO org.apache.camel.impl.engine.AbstractCamelContext - Apache Camel 3.4.2 (camel-1) is starting
[main] INFO org.apache.camel.impl.engine.AbstractCamelContext - StreamCaching is not in use. If using streams then its recommended to enable stream caching. See more details at http://camel.apache.org/stream-caching.html
[main] INFO org.apache.camel.impl.engine.AbstractCamelContext - Using HealthCheck: camel-health
[main] INFO org.apache.camel.impl.engine.InternalRouteStartupManager - Route: route1 started and consuming from: aws2-s3://mybucket
[main] INFO org.apache.camel.impl.engine.InternalRouteStartupManager - Route: route2 started and consuming from: direct://httproute
[main] INFO org.apache.camel.impl.engine.AbstractCamelContext - Total 3 routes, of which 3 are started
[main] INFO org.apache.camel.impl.engine.AbstractCamelContext - Apache Camel 3.4.2 (camel-1) started in 1.496 seconds
[Camel (camel-1) thread #1 - aws2-s3://mybucket] INFO consuming - Consumer Fired!
[Camel (camel-1) thread #1 - aws2-s3://mybucket] INFO route1 - Replay Message Sent to file:s3out input_2020-07-01-07-20-31-796_0.csv.gz
[Camel (camel-1) thread #1 - aws2-s3://mybucket] ERROR org.apache.camel.processor.errorhandler.DefaultErrorHandler - Failed delivery for (MessageId: ID-DGHANTA-Mac-1599955541230-0-1 on ExchangeId: ID-DGHANTA-Mac-1599955541230-0-1). Exhausted after delivery attempt: 1 caught: java.util.zip.ZipException: Not in GZIP format
Message History (complete message history is disabled)
---------------------------------------------------------------------------------------------------------------------------------------
RouteId ProcessorId Processor Elapsed (ms)
[route1 ] [route1 ] [from[aws2-s3://mybucket?amazonS3Client=%23] [ 250]
...
[route1 ] [unmarshal1 ] [unmarshal[org.apache.camel.model.dataformat.GzipDataFormat@d41f816] ] [ 0]
Stacktrace
---------------------------------------------------------------------------------------------------------------------------------------
java.util.zip.ZipException: Not in GZIP format
at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:165)
at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:79)
at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:91)
at org.apache.camel.dataformat.deflater.GzipDeflaterDataFormat.unmarshal(GzipDeflaterDataFormat.java:62)
at org.apache.camel.support.processor.UnmarshalProcessor.process(UnmarshalProcessor.java:64)
at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$SimpleTask.run(RedeliveryErrorHandler.java:395)
at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:148)
at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:60)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:147)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:286)
at org.apache.camel.component.aws2.s3.AWS2S3Consumer.processBatch(AWS2S3Consumer.java:200)
at org.apache.camel.component.aws2.s3.AWS2S3Consumer.poll(AWS2S3Consumer.java:113)
at org.apache.camel.support.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:187)
at org.apache.camel.support.ScheduledPollConsumer.run(ScheduledPollConsumer.java:106)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Regards,
Dinesh
{{}}