Uploaded image for project: 'Camel'
  1. Camel
  2. CAMEL-13624

Pollenrich issue while consuming second file with the same name

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Invalid
    • Affects Version/s: 2.10.4, 2.16.3
    • Fix Version/s: None
    • Component/s: came-core, camel-sftp
    • Labels:
      None
    • Estimated Complexity:
      Unknown

      Description

      Pollenrich issue while consuming second file with the same name

      We have implemented a logic where a checksum will be compared during poll enrich using aggregationStrategy.

      Below is the entire flow of this logic

      1. Poll xls file from sftp location. Lets say the name of the file will be sample.xls
      ftp.poll.options=&delay=ftp.bt_in.delay&initialDelay=ftp.bt_in.delay\
      &binary=true&readLock=change\
      &preMove=.inprogress&moveFailed=../.failed&delete=true\
      &throwExceptionOnConnectFailed=true&pollStrategy=#ftpPollStrategy\
      &localWorkDirectory=ftp.localWorkDirectory\
      &strictHostKeyChecking=sftp.strictHostKeyChecking&knownHostsFile=sftp.knownHostsFile\
      &runLoggingLevel=DEBUG&disconnect=true

      2. Poll subsequent sha file where the name will be sample.xls.sha512
      ftp.sha512.poll.options=&delay=2000&initialDelay=2000\
      &binary=true&readLock=change\
      &moveFailed=.failed&delete=true\
      &throwExceptionOnConnectFailed=true&pollStrategy=#ftpPollStrategy\
      &strictHostKeyChecking=sftp.strictHostKeyChecking&knownHostsFile=sftp.knownHostsFile\
      &runLoggingLevel=DEBUG
      3. Polling sha file using pollenrich is as below

      from("direct:" + routeId)
      .id(routeId)
      .onCompletion()
      .process(new ShutDownProcessor())
      .end()
      .log(LoggingLevel.INFO,
      "Polling SHA512 checksum for transfered file '${in.header.CamelFileName}' with timeout " + timeout + " ms")
      .log(LoggingLevel.DEBUG,"SHA512 dynamic polling route details: enrichUri: " + this.enrichUri)
      .convertBodyTo(byte[].class, "UTF-8")
      .pollEnrich(enrichUri, timeout, MY_AGGREGATION_STRATEGY)
      .to(outUri);
      }
      Where
      enrichUri - "source.ftp.endpointftp.sha512.poll.options" + "&fileName=" + fileName + ".sha512"
      timeout - 30000
      MY_AGGREGATION_STRATEGY - Strategy to compare old checksum

      4. Aggregator logic has been implemeted as below.

      public Exchange aggregate(Exchange oldExchange, Exchange newExchange) {
      if (newExchange == null)

      { log.error("No sftp file found for the input file!"); throw new RuntimeException("No sftp file found for the input file!"); }

      Message sha512Msg = newExchange.hasOut() ? newExchange.getOut() : newExchange.getIn();
      Message incomingMsg = oldExchange.hasOut() ? oldExchange.getOut() : oldExchange.getIn();

      try {
      String incomingSHA512 = fetch sha of received file
      String computedSHA512 = fetch sha of computed file

      if(incomingSHA512.equals(computedSHA512))

      { log.info("SHA512 check for file '" + oldExchange.getIn().getHeader("CamelFileName", String.class) + "' successful."); return oldExchange; }

      else

      { throw new RuntimeException("Computed SHA512 checksum '" + computedSHA512 + "' does not match with provided SHA512 checksum '" + incomingSHA512 + "'"); }

      } catch (Exception e)

      { log.error("Error while processing SHA512 checksum of the passing file: " + e.getMessage()); throw new RuntimeException(e); }

      Now the issue which we are facing is

      1. First time if we introduce sample.xls and corresponding sha file where checksum doesnt match, then it will throw an exception and the file will be moved to failed folder. This is an expected behavior and works fine.
      2. If we introduce the same file again with the same name, then sample.xls file will be consumed, but corresponding sha file will not be consumed. The file is present in the location but still it will landeup in "No sftp file found for the input file!" which we have logged in aggregate method.
      3. Those file will consume if i restart the server.

      We have tried below
      1. We are using camel 2.10.4 and tried with 2.16.3.
      2. Adding idempotent=false in ftp.sha512.poll.options.
      3. Changing timer and keep it for quite a long time.

      But still facing same issue.

      Are we missing anything here?

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              sharathchitra Sharath
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: