Description
I'm using ExecuteSQLRecord processor to query from PostgreSQL. A 'select' query I'm using can return more than 60 million rows. I have configured the fetch size to 1000 to avoid fetching all the data into memory at once.
But when the processor is started, heap starts to grow very fast. I have configured to NIFI to have 50GB heap size and even that amount is filled within minutes. When the heap is filled Garbage collector tries to clean the heap blocking other threads.
It seems like NIFI loads all the data to memory even though fetch size is set to 1000. I have attached the NIFI's GC log here with this ticket for reference.
Attachments
Attachments
Issue Links
- relates to
-
NIFI-12010 Handle auto-commit and commit based on driver capabilities in SQL components
-
- Resolved
-
- links to
