Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-3990

Reduce excessive garbage collection caused by record readers

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.3.0
    • Extensions
    • None

    Description

      Record Readers and writers use the DataTypeUtils class to coerce an object of one type to another type. Often, these methods are called with a Supplier<DateFormat> to avoid creating DateFormat objects when not necessary. However, the Supplier is typically created inline with a lambda when it doesn't need to be, and this creates significant pressure on the garbage collector.

      Also, the JSON Reader performs a significant amount of string concatenation when it encounters a Map, in order to generate the field name including the Map Key. This is done only so that the map key is included in the error message if one occurs. However, I think this should be avoided, since the error message already indicates the name of the map field and the value that could not be coerced. Avoiding the concatenation there will cut down significantly on GC pressure, as evidenced by object creation profiling.

      Attachments

        Issue Links

          Activity

            People

              markap14 Mark Payne
              markap14 Mark Payne
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: