[NIFI-2195] Improve sophistication of how nodes detect and handle differences between local and cluster flows

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.0.0
• Component/s: Core Framework
• Assignee: Unassigned
• Reporter: Mark Payne
• Created: 07/Jul/16 21:31
• Updated: 13/Oct/16 13:05

Description

Currently, when a node joins a cluster, it uses the flow's "Fingerprint" to determine whether or not its flow matches the cluster flow. If not, it will not join the cluster. There are two main drivers for this. First, we want to ensure that we don't overwrite the flow with a completely different flow (though we want to allow for minor differences between the flow, such as the location of a processor on the graph). Secondly, we want to ensure that there is no data loss.

We can improve the sophistication of this and allow a node to join the cluster if there are no 'substantive' differences. It is critical that a node not inherit a flow that is missing some Connection, if that Connection has data queued up, as doing so would result in data loss.

We should be smarter, though, about which things really matter. For instance, if the name of a Processor is different, the node should simply inherit the name of the Processor from the cluster, instead of refusing to join.

If any changes to the flow are made when joining a cluster, though, we should ensure that we back up the flow before changing it.

Issue Links

• is related to: NIFI-1966 Address Issue of Nodes Disagreeing About Flow Contents on Startup