Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-9236

Update UI for Clustered Connections

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.13.2
    • None
    • Core UI
    • None

    Description

      I have noticed that when I set the back pressure threshold to 10k flowfiles in a clustered setup, it will turn red and display in the tooltip: "Queue: 100% full" when a single node reaches this limit. For instance, if I have a simple flow:

      3 Node cluster
      Generate Flowfile (run only on primary node) -> Connection with 10k backpressure queue size -> Disabled Processor 

      It will run until the primary node hits the 10k backpressure limit and show the connection as red even though the other nodes in the cluster have 0 flowfiles. Now If I update the Generate Flowfile to run on all nodes, it will be able to output up to 30k flowfiles, even though the backpressure already has said it was 100% full. Since the UI does not show which node has reached the limit and that it continues to let in more flowfiles past the 10k configured limit, it can quickly become confusing. Additionally, the UI does not indicate that the clustered maximum threshold is actually the maximum threshold multiplied by the number of nodes. 

      In a clustered setup, it would be helpful if we could see the breakout of flowfiles in connections per nifi. This would allow us to quickly identify nodes in the cluster that are overworked/slow and nodes that are under-utilized. Also, it could say something like "Queue: 100% full for 1/3 nodes" or some other visual indicator that it is applying backpressure for some nodes, but not for the entire cluster. Additionally, it could indicate the true maximum object/size threshold in the clustered connection.

      Attachments

        Activity

          People

            Unassigned Unassigned
            aheys Andrew Heys
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated: