Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-1669

MergeContent does not always honor the 'Maximum number of Bins' property

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.6.0
    • Component/s: Extensions
    • Labels:
      None

      Description

      MergeContent exposes a "Maximum number of Bins" property. However, the way in which this property is used limits only the maximum number of 'in progress' bins. Once a bin has been declared 'ready' (i.e., the FlowFiles have been grouped together and are just waiting to be written out), that 'ready bin' is not counted toward the maximum number of allowable bins.

      As a result, if we have a very high volume of FlowFiles coming in, we could end up with MergeContent creating many bins and buffering potentially millions of FlowFiles, exhausting Java Heap and preventing backpressure and age-off from kicking in.

        Attachments

          Activity

            People

            • Assignee:
              markap14 Mark Payne
              Reporter:
              markap14 Mark Payne
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: