Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-1329

ControlRate processor should not be penalizing FlowFiles

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.4.1
    • Fix Version/s: 0.6.0
    • Component/s: Core Framework
    • Labels:
      None

      Description

      The controlRate processor currently penalizes all FlowFiles still on input queue if flow rate has been reached. Those penalized FlowFiles will not be processed by the ControlRate processor again until that penalization has expired causing an impact on the desired controlled throughput of the processor.

      Example ControlRate processor setup:
      Penalty duration left at default 30 sec
      Strategy set to FlowFile count
      Max rate set at 100
      time duration set to 1 sec

      Let say the input queue to this processor has 2000 files dumped in it. The ControlRate processor will read all 2000 files, allow 100 through and penalize the other 1900 fro 30 seconds. Assuming no other files enter queue at this point, after 30 seconds another 100 will pass and 1800 will be penalized again. This pattern continues....

      You can work around this defect by changing the penalization default from 30 to say 1 sec.

      Why is the "In" stat reflecting reading all files on input queue when only 100 files are actually passing through processor? <-- this leads to confusion for the users.
      Why is penalty being applied here?

        Attachments

          Activity

            People

            • Assignee:
              mosermw Michael Moser
              Reporter:
              msclarke Matthew Clarke
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: