[CAMEL-13624] Pollenrich issue while consuming second file with the same name

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 2.10.4, 2.16.3
• Component/s: came-core, camel-sftp
• Estimated Complexity: Unknown
• Assignee: Unassigned
• Reporter: Sharath
• Created: 07/Jun/19 06:39
• Updated: 07/Jun/19 06:48
• Resolved: 07/Jun/19 06:47

Description

Pollenrich issue while consuming second file with the same name

We have implemented a logic where a checksum will be compared during poll enrich using aggregationStrategy.

Below is the entire flow of this logic

1. Poll xls file from sftp location. Lets say the name of the file will be sample.xls
ftp.poll.options=&delay=ftp.bt_in.delay&initialDelay=ftp.bt_in.delay\
&binary=true&readLock=change\
&preMove=.inprogress&moveFailed=../.failed&delete=true\
&throwExceptionOnConnectFailed=true&pollStrategy=#ftpPollStrategy\
&localWorkDirectory=ftp.localWorkDirectory\
&strictHostKeyChecking=sftp.strictHostKeyChecking&knownHostsFile=sftp.knownHostsFile\
&runLoggingLevel=DEBUG&disconnect=true

2. Poll subsequent sha file where the name will be sample.xls.sha512
ftp.sha512.poll.options=&delay=2000&initialDelay=2000\
&binary=true&readLock=change\
&moveFailed=.failed&delete=true\
&throwExceptionOnConnectFailed=true&pollStrategy=#ftpPollStrategy\
&strictHostKeyChecking=sftp.strictHostKeyChecking&knownHostsFile=sftp.knownHostsFile\
&runLoggingLevel=DEBUG
3. Polling sha file using pollenrich is as below

from("direct:" + routeId)
.id(routeId)
.onCompletion()
.process(new ShutDownProcessor())
.end()
.log(LoggingLevel.INFO,
"Polling SHA512 checksum for transfered file '${in.header.CamelFileName}' with timeout " + timeout + " ms")
.log(LoggingLevel.DEBUG,"SHA512 dynamic polling route details: enrichUri: " + this.enrichUri)
.convertBodyTo(byte[].class, "UTF-8")
.pollEnrich(enrichUri, timeout, MY_AGGREGATION_STRATEGY)
.to(outUri);
}
Where
enrichUri - "source.ftp.endpointftp.sha512.poll.options" + "&fileName=" + fileName + ".sha512"
timeout - 30000
MY_AGGREGATION_STRATEGY - Strategy to compare old checksum

4. Aggregator logic has been implemeted as below.

public Exchange aggregate(Exchange oldExchange, Exchange newExchange) {
if (newExchange == null)

{ log.error("No sftp file found for the input file!"); throw new RuntimeException("No sftp file found for the input file!"); }

Message sha512Msg = newExchange.hasOut() ? newExchange.getOut() : newExchange.getIn();
Message incomingMsg = oldExchange.hasOut() ? oldExchange.getOut() : oldExchange.getIn();

try {
String incomingSHA512 = fetch sha of received file
String computedSHA512 = fetch sha of computed file

if(incomingSHA512.equals(computedSHA512))

{ log.info("SHA512 check for file '" + oldExchange.getIn().getHeader("CamelFileName", String.class) + "' successful."); return oldExchange; }

else

{ throw new RuntimeException("Computed SHA512 checksum '" + computedSHA512 + "' does not match with provided SHA512 checksum '" + incomingSHA512 + "'"); }

} catch (Exception e)

{ log.error("Error while processing SHA512 checksum of the passing file: " + e.getMessage()); throw new RuntimeException(e); }

Now the issue which we are facing is

1. First time if we introduce sample.xls and corresponding sha file where checksum doesnt match, then it will throw an exception and the file will be moved to failed folder. This is an expected behavior and works fine.
2. If we introduce the same file again with the same name, then sample.xls file will be consumed, but corresponding sha file will not be consumed. The file is present in the location but still it will landeup in "No sftp file found for the input file!" which we have logged in aggregate method.
3. Those file will consume if i restart the server.

We have tried below
1. We are using camel 2.10.4 and tried with 2.16.3.
2. Adding idempotent=false in ftp.sha512.poll.options.
3. Changing timer and keep it for quite a long time.

But still facing same issue.

Are we missing anything here?