Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Done
-
1.7.1
-
None
-
None
Description
Allow a node in the cluster to be offloaded, rebalancing flowfiles on that node to the other active nodes.
Only nodes that are DISCONNECTED can be transitioned to the OFFLOADING state.
OFFLOADING nodes will transition to the OFFLOADED state once all flowfiles have been rebalanced to other connected nodes in the cluster.
OFFLOADING nodes that remain in the OFFLOADING state (due to errors encountered while offloading) can be reconnected to the cluster by restarting NiFi on the node.
OFFLOADED nodes can be reconnected to the cluster by issuing a connection request via the UI/CLI, or restarting NiFi on the node.
OFFLOADED nodes can be deleted from the cluster.
Once a node is OFFLOADED, it will behave like a DISCONNECTED node.
OFFLOADING a node:
- stops all processors
- terminates all processors
- stops transmitting on all remote process groups
- rebalances flowfiles to other connected nodes in the cluster (via the work done in
NIFI-5516)
The steps to decommission a node and remove it from the cluster are:
- Send request to disconnect the node
- Once disconnect completes, send request to offload the node.
- Once offload completes, send request to delete node.
- Once the delete request has finished, the NiFi service on the host can be stopped/removed.
When an error occurs and the node can not complete offloading, the user can:
- Send request to delete the node from the cluster
- Diagnose why the node had issues with the offload (out of memory, no network connection, etc) and address the issue
- Restart NiFi on the node to so that it will reconnect to the cluster
- Go through the steps to decommission a node
The OFFLOADING request is idempotent.
Toolkit CLI commands for retrieving a single node, list of nodes, and connecting/disconnecting/offloading/deleting nodes have been added.
The cluster table UI has an icon to initiate the OFFLOADING of a DISCONNECTED node.
Similar to the client sending PUT request with a DISCONNECTING message to cluster/nodes/{id}, an OFFLOADING message can be sent as a PUT request to the same URI to initiate an OFFLOAD for a DISCONNECTED node.
