Uploaded image for project: 'Camel'
  1. Camel
  2. CAMEL-12211

Idempotent on camel 2.20.0 and 2.20.1 allows duplication

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 2.20.0, 2.20.1
    • 2.21.0
    • camel-core
    • None
    • Reproduced on Linux and OSX

    • Unknown

    Description

      On a project with at least two routes and idempotent repository on both source endpoints and another on the route to avoid duplication when the number of exchanges goes beyond certain size the target starts to get duplicates. 

      Have created a sample project where the problem can be reproduced on https://github.com/pampanz/camel-idempotent-issue

      Details on how to use the project and reproduce the issue are on the README.md of the project.

       

      The route setup in the project is

      from("file://./fromSource?delay=5000&idempotent=true&idempotentKey=\$simple

      {[file:name|file:///name]}

      &idempotentRepository=#myRepo")
      .autoStartup(true)
      .routeId("source")
      .log("Received file [\${file:name}], proceed to move to target")
      .to("direct:dedup")

      from("file://./fromSource2?delay=5000&idempotent=true&idempotentKey=\$simple

      {[file:name|file:///name]}

      &idempotentRepository=#myRepo")
      .autoStartup(true)
      .routeId("source2")
      .log("Received file [\${file:name}], proceed to move to target")
      .to("direct:dedup")

      from("direct:dedup")
      .routeId("Deduping and moving")
      .log("Deduping file [\${file:name}]")
      .idempotentConsumer(simple('${file:name}'), FileIdempotentRepository.fileIdempotentRepository(new File("dedup.idempotent.file.store"), 100000, 1000000))
      .setHeader("CamelFileName",simple('${file:name}-${date:now:yyyyMMddhhmmss}'))
      .log("Final file to the target is [\${file:name}]")
      .to("file://./toTarget")

       

      If more than a thousand files are available on both sources once the server finish processing them the same files are copied over then camel 2.20.0 and 2.20.1 starts creating duplicates on the target folder. The same DOES NOT happen on 1.19.4, 1.19.0. We haven't tried but are pretty sure it didn't happen on previous versions either.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            davsclaus Claus Ibsen
            pampanz Pablo Sosa
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment