Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-8605

ExecuteSQLRecord processor consumes a large heap volume when use with PostgreSQL JDBC driver

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.16.0, 1.15.3
    • None

    Description

      I'm using ExecuteSQLRecord processor to query from PostgreSQL. A 'select' query I'm using can return more than 60 million rows. I have configured the fetch size to 1000 to avoid fetching all the data into memory at once.

      But when the processor is started, heap starts to grow very fast. I have configured to NIFI to have 50GB heap size and even that amount is filled within minutes. When the heap is filled Garbage collector tries to clean the heap blocking other threads. 

      It seems like NIFI loads all the data to memory even though fetch size is set to 1000. I have attached the NIFI's GC log here with this ticket for reference.

      Attachments

        1. GC.LOG
          7 kB
          Vibhath Arunapriya Ileperuma

        Issue Links

          Activity

            People

              Vibhath Vibhath Arunapriya Ileperuma
              Vibhath Vibhath Arunapriya Ileperuma
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 50m
                  1h 50m