Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-2987

RPG does not do load-balancing well when getting FlowFiles from output ports

Agile BoardAttach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.0.0
    • Fix Version/s: None
    • Component/s: Core Framework
    • Labels:
      None

      Description

      When a RPG connects to a destination system's output port, it retrieves every FlowFile queued at the time of the connection. If the source system with the RPG is a NiFi cluster, only one node in the cluster receives all the FlowFiles from that output port. If there is a steady stream of FlowFiles to the output port, there is still no true balanced delivery of data.

      We need to be able to limit the number of FlowFiles per connection when the source of the FlowFiles is an output port.

      When the destination system with the output port is a cluster and the source system is a cluster with a RPG, the first node to connect will pull all data from the node with the highest queue. Next node will pull from next highest queued destination node and so on. There is also no guarantee that avery node in the source cluster will get any data.

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              ijokarumawak Koji Kawamura
              Reporter:
              msclarke Matthew Clarke

              Dates

              • Created:
                Updated:

                Issue deployment