[NIFI-5702] FlowFileRepo should not discard data (at least not by default)

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.7.1, 1.9.2
• Fix Version/s: 1.12.0
• Assignee: Mark Payne
• Reporter: Brandon Rhys DeVries
• Created: 15/Oct/18 16:40
• Updated: 06/Aug/20 14:56
• Resolved: 06/Aug/20 14:56

Description

The WriteAheadFlowFileRepository currently discards data it cannot find a queue for.  Unfortunately, we have run in to issues where, when rejoining a node to a cluster, the flow.xml.gz can go "missing".  This results in the instance creating a new, empty, flow.xml.gz and then continuing on... and not finding queues for any of its existing data, dropping it all.  Regardless of the circumstances leading to an empty (or unexpectedly modified) flow.xml.gz, dropping data without user input seems less than ideal. 

Internally, my group has added a property "....remove.orphaned.flowfiles.on.startup", defaulting to "false".  On startup, rather than silently dropping data, the repo will throw an exception preventing startup.  The operator can then choose to either "fix" any unexpected issues with the flow.xml.gz, or they can set the above property to "true" which restores the original behavior allowing the system to be restarted.  When set to "true" this property also results in a warning message indicating that in this configuration the repo can drop data without (advance) warning.  

 

 

[1] https://github.com/apache/nifi/blob/support/nifi-1.7.x/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/WriteAheadFlowFileRepository.java#L596

Issue Links

• links to: GitHub Pull Request #4454