Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-8435

PutKudu 1.13.2 Memory Leak

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 1.13.2
    • 1.14.0
    • Extensions
    • NiFi 1.13.2, 8-Node Cluster running on CentOS 7, Kudu 1.10.0

    Description

      We just upgraded from NiFi 1.11.4 to 1.13.2 and faced a huge issue with PutKudu.

      PutKudu on the 1.13.2 eats up all the heap memory and garbage collection can't anymore free up the memory. We allow Java to use 31GB memory and as you can see with NiFi 1.11.4 it will be used like it should with GC. However with NiFi 1.13.2 with our actual load it fills up the memory relatively fast. Manual GC via visualvm tool didn't help at all to free up memory.

       

      Visual VM shows the following culprit:

      The bytes array shows millions of char data which isn't cleaned up. In fact here 14,9GB memory (heapdump has been taken after a while of full load). If we check the same on NiFi 1.11.4, the bytes array is nearly empty, around a few hundred MBs.

      As you could imagine we can't upload the heap dump as currently we have only productive data on the system. But don't hesitate to ask questions about the heapdump if you need more information.

      I haven't done any screenshot of the processor config, but I can do that if you wish (we are back to NiFi 1.11.4 at the moment). 

      Attachments

        1. visualvm_total_bytes_used.png
          124 kB
          Josef Zahner
        2. visualvm_bytes_detail_view.png
          189 kB
          Josef Zahner
        3. Screenshot 2021-04-20 at 14.27.11.png
          57 kB
          Peter Gyori
        4. putkudu_processor_config.png
          321 kB
          Josef Zahner
        5. kudu_inserts_per_sec.png
          76 kB
          Josef Zahner
        6. grafana_heap_overview.png
          554 kB
          Josef Zahner

        Issue Links

          Activity

            People

              pgyori Peter Gyori
              jzahner Josef Zahner
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1.5h
                  1.5h