[NIFI-1316] Allow DetectDuplicate to only detect and not cache

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.4.1
• Fix Version/s: 0.5.0
• Assignee: Joe Percivall
• Reporter: Joe Percivall
• Created: 21/Dec/15 16:37
• Updated: 21/Jan/16 04:29
• Resolved: 12/Jan/16 04:13

Description

Working on a Webcrawler template/documentation I find myself wanting to have a pair of detect duplicate processors. One of which does the typical check, cache and remove if duplicate. The other I want to only check and remove if Dup (don't add them to the cache in that processor).

The use-case being I want to add URLs to the cache after being successfully reached by the InvokeHttp processor. I also would like to check for urls that were successfully reached before even sending them to the InvokeHttp processor but I don't want to add to the cache before InvokeHttp because they might not successfully hit the URL.

I attached the template to the ticket. You can see how the DetectDuplicate going into InvokeHttp should only check for duplicates and not cache them (because the URL hasn't been successfully hit yet).

Ideally this improvement would only require a configuration option added to the processor which gives the option whether or not to cache.