Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-5946

Hive3Streaming memory leak

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 1.8.0
    • Fix Version/s: 1.9.0
    • Component/s: Extensions
    • Labels:
      None
    • Environment:
      Rhel 7.5
      Open jdk 1.8
      NiFi 1.8
      HDP 3

      Description

      Our NiFi instance has been leaking memory since started using Hive3Streaming processor and now we are seeing memory usage of 60+ GB. Looking at the processor code it seems like we are:

      1. Creating a new StreamingConnection with each flow file
      2. Registering a shutdown hook via <code>ShutdownHookManager.addShutdownHook</code> that should close the connection
      3. Closing the connection and not removing the shutdown hook.

      The Runnable captures the connection and conf objects, and since our NiFi instance has been running continuously and the JVM hasn't shutdown (3 weeks +) the ShutdownHookManager is still holding reference to the connections created since causing the memory leak.

        Attachments

        1. nifi_memory_leak.png
          18 kB
          Gideon Korir

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                korir Gideon Korir
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: