Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-2848

Queues aren't fairly drained when leading to a single component

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0, 0.7.0
    • Fix Version/s: 1.1.0
    • Component/s: Core Framework
    • Labels:
      None

      Description

      Consider the scenario where multiple queues lead to a single component and all of them are full due to back pressure. With the attached template, it is easily observable that once a single queue starts to drain due to relieved back pressure, it will continue to drain as long as it has incoming flow files. This means that if there's a constant flow of incoming flow files to this queue, the other queues will never be drained (at least, that's my theory based on several hours of observation).

      To reproduce this:

      1. Load the template into NiFi 1.0.0
      2. Play all three GenerateFlowFile processors, but not the UpdateAttribute processor (this simulates backpressure). Wait until each queue has 1,000 flow files (max backpressure)
      3. Stop the GenerateFlowFile processors, and play the UpdateAttribute processor (this relieves the backpressure)
      4. Observe which queue has started to drain, and start its GenerateFlowFile processor
      5. Observe that the other two queues remain full indefinitely, while the draining queue continues to replenish and be drained indefinitely

        Attachments

        1. dominate_2.png
          178 kB
          Toivo Adams
        2. dominate_1.png
          177 kB
          Toivo Adams
        3. queue_drain.png
          200 kB
          Toivo Adams
        4. Backpressure_prioritization_test.xml
          12 kB
          Joseph Gresock

          Issue Links

            Activity

              People

              • Assignee:
                pvillard Pierre Villard
                Reporter:
                jgresock Joseph Gresock
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: