Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-5466

Improve efficiency of RingBufferEventRepository

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.8.0
    • Component/s: Core Framework
    • Labels:
      None

      Description

      The implementation of the FlowFileEventRepository (i.e., the RingBufferEventRepository) is implemented in such a way that 300 buckets are kept for each component. Each bucket holds a set of 16+ stats (bytes read, bytes written, flowfiles in, flowfiles out, etc. as well as counters)

      Each time that a user refreshes stats, we calculate the 5-minute window for each component by iterating over all 300 buckets, then summing together the values of the stats. So, for every 1,000 processors, we have 1000 * 300 * 16 = 4.8 MM calculations being performed each time that the user refreshes.

      We can improve this significantly. To do this, we can keep a 'running tally' for each stat. Every time that we update a bucket in the FlowFileEventRepository, we should similarly update the running tally. Each time that we 'replace' a bucket in the FlowFileEventRepository, we should subtract from the running tally the values in the bucket, in addition to adding in the new value. This would result in amortizing the cost of calculation over the 5-minute period and means that every time that we get the processor stats we could do so by simply performing a 'get' operation without any calculations.

      But wait, there's more! The EventSumValue class has an add(FlowFileEvent) method. If the EventSumValue were to simply keep a boolean value indicating whether or not it was empty, then when adding to an empty bucket we could avoid the calculations all together, and when replacing an empty bucket we could avoid the calculation against the running tally as well.

      Similarly, in the VolatileComponentStatusRepository, when we perform a 'capture' we should detect any 'empty' stats objects and use a single static instance for this, which would significantly reduce the amount of heap that is used to store metrics.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                markap14 Mark Payne
                Reporter:
                markap14 Mark Payne
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: