Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Duplicate
-
2.20.0, 2.20.1
-
None
-
Reproduced on Linux and OSX
-
Unknown
Description
On a project with at least two routes and idempotent repository on both source endpoints and another on the route to avoid duplication when the number of exchanges goes beyond certain size the target starts to get duplicates.
Have created a sample project where the problem can be reproduced on https://github.com/pampanz/camel-idempotent-issue
Details on how to use the project and reproduce the issue are on the README.md of the project.
The route setup in the project is
from("file://./fromSource?delay=5000&idempotent=true&idempotentKey=\$simple
{[file:name|file:///name]}&idempotentRepository=#myRepo")
.autoStartup(true)
.routeId("source")
.log("Received file [\${file:name}], proceed to move to target")
.to("direct:dedup")
from("file://./fromSource2?delay=5000&idempotent=true&idempotentKey=\$simple
{[file:name|file:///name]}&idempotentRepository=#myRepo")
.autoStartup(true)
.routeId("source2")
.log("Received file [\${file:name}], proceed to move to target")
.to("direct:dedup")
from("direct:dedup")
.routeId("Deduping and moving")
.log("Deduping file [\${file:name}]")
.idempotentConsumer(simple('${file:name}'), FileIdempotentRepository.fileIdempotentRepository(new File("dedup.idempotent.file.store"), 100000, 1000000))
.setHeader("CamelFileName",simple('${file:name}-${date:now:yyyyMMddhhmmss}'))
.log("Final file to the target is [\${file:name}]")
.to("file://./toTarget")
If more than a thousand files are available on both sources once the server finish processing them the same files are copied over then camel 2.20.0 and 2.20.1 starts creating duplicates on the target folder. The same DOES NOT happen on 1.19.4, 1.19.0. We haven't tried but are pretty sure it didn't happen on previous versions either.