Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-3559

Improve S2S load-balancing

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.1.1
    • Fix Version/s: None
    • Component/s: Core Framework
    • Labels:
      None

      Description

      The current implementation of S2S sends data continuously to the destination NiFi node for 0.5 seconds before closing the connection and opening a new connection to another node.

      When the source FlowFile are all very small (0 byte in case of list based processors), the entire queue can end up getting sent to only one of the target NiFi cluster nodes.

      Another common use case for S2S is to have a RPG pointed back at same cluster where the RPG was added. Since FlowFiles are likely to transfer to the same node where the data originates (Think Primary node data redistribution within a cluster) much faster then transfers to other nodes, the primary node is likely to always end up with more FlowFiles then any other node.

      There needs to be an additional load-balancing strategy that compliments the existing 0.5 second to improve upon the load-balancing in such cases. The RPG know how many target nodes there are and how many FlowFiles exist in the queue at run time, so perhaps using that info to more even split the queue amongst all nodes smartly would help.

      This is related to existing Jira: NiFI-2987

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                msclarke Matthew Clarke
              • Votes:
                1 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: