[NIFI-2825] Remote Process Group should use num of queued flow files in remote nodes to distribute load

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.1.0
• Component/s: Core Framework
• Labels: Cluster
• Environment: Clustered
• Assignee: Koji Kawamura
• Reporter: Koji Kawamura
• Created: 27/Sep/16 00:53
• Updated: 03/Oct/16 17:59
• Resolved: 03/Oct/16 17:59

Description

A RPG (Site-to-Site client) should take into account of the number of queued flow files in each remote cluster nodes, to calculate the load distribution balance among target nodes. So that it can send more data to nodes those have less queued files, or receive more data from nodes those have more queued files.

However, after the clustering mechanism has changed since 1.0, this load distribution logic is not handled appropriately. Site-to-Site remote component always returns flow file count as 0. So RPG distribute load evenly, even if a remote cluster has unbalanced load among nodes.

We need to fix the remote node API to expose queued flow file count correctly.

Issue Links

• links to: GitHub Pull Request #1084