[NIFI-9475] Provide Framework-Level Retries for NiFi Relationships

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.16.0
• Assignee: Timea Barna
• Reporter: Timea Barna
• Created: 10/Dec/21 12:58
• Updated: 31/May/22 09:03
• Resolved: 15/Feb/22 20:40

Description

For a processor that routes to failure, in NiFi I have 3 options:

1) Loop 'failure' back to the processor and retry forever

2) Auto-terminate or route the data elsewhere without any retries

3) Build a complex retry loop that creates or updates an attribute, routes on that attribute to determine whether or not we've exceeded the maximum number of retries, and if not penalizes the FlowFile (typically with a Script processor) and then routes back to try the FlowFile again.

This provides a poor user experience and poor performance and results in very complex flows that are difficult to understand and maintain.

As a user, when I configure a Processor, I should be able to indicate that for Relationship ABC, I want to retry the FlowFile 5 times. Then, if the Processor transfers a FlowFile to that relationship, it should instead transfer the FlowFile back to its original queue and increment the transfer count for that relationship from 0 to 1 and penalize the FlowFile. If the Processor processes it again and transfers it to the same Relationship again, it'll again put it back to the original queue and increment the count to 2. Only when the data is routed to that Relationship 5 times (because that's the configured retry count in this example) should the FlowFile actually be transferred to that corresponding connection(s) or auto-terminated, as the flow is configured.

Issue Links

• relates to: NIFI-9756 Add documentation for framework-level retry in Processors
• links to: GitHub Pull Request #5593