Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-1316

Allow DetectDuplicate to only detect and not cache

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.4.1
    • Fix Version/s: 0.5.0
    • Component/s: None
    • Labels:
      None

      Description

      Working on a Webcrawler template/documentation I find myself wanting to have a pair of detect duplicate processors. One of which does the typical check, cache and remove if duplicate. The other I want to only check and remove if Dup (don't add them to the cache in that processor).

      The use-case being I want to add URLs to the cache after being successfully reached by the InvokeHttp processor. I also would like to check for urls that were successfully reached before even sending them to the InvokeHttp processor but I don't want to add to the cache before InvokeHttp because they might not successfully hit the URL.

      I attached the template to the ticket. You can see how the DetectDuplicate going into InvokeHttp should only check for duplicates and not cache them (because the URL hasn't been successfully hit yet).

      Ideally this improvement would only require a configuration option added to the processor which gives the option whether or not to cache.

        Attachments

        1. 0002-NIFI-1316.patch
          17 kB
          Joe Percivall
        2. 0001-NIFI-1316-adding-option-to-DetectDuplicate-to-not-ca.patch
          10 kB
          Joe Percivall
        3. WebCrawler.xml
          133 kB
          Joe Percivall

          Activity

            People

            • Assignee:
              jpercivall Joe Percivall
              Reporter:
              jpercivall Joe Percivall
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: