Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-7114

DistributedMapCacheClient/Server not closing resources leading to exhausted file handles

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 1.10.0, 1.11.0
    • 1.12.0, 1.11.3
    • Configuration
    • None
    • Amazon EC2 running either Amazon Linux 2 or Ubuntu 18.04.
      NiFi has been installed with no change to any configuration file.

    Description

      Since at least version 1.10 NiFi stopped closing file handles. It opens circa 500 files per hour (measured using lsof) without any apparent limit until it crashes due to too many open files.

       

      Increasing the computer open file limit is not a solution since NiFi will still crash, it'll only take longer to do so.

      Attachments

        1. flow.xml.gz
          2 kB
          Paul Kelly
        2. fifocounts.txt
          1 kB
          Paul Kelly
        3. reproduction.zip
          48.04 MB
          Paul Kelly
        4. openFiles.xlsx
          65 kB
          Vinicius Zolin
        5. lsof.zip
          7.87 MB
          Vinicius Zolin
        6. destination.xml
          145 kB
          Vinicius Zolin
        7. source.xml
          14 kB
          Vinicius Zolin
        8. lsofBefore.log
          1.12 MB
          Vinicius Zolin
        9. lsofAfter.log
          1.13 MB
          Vinicius Zolin
        10. lsof.log
          1.12 MB
          Vinicius Zolin

        Issue Links

          fixes

          Bug - A problem which impairs or prevents the functions of the product. NIFI-6790 Load-balanced queues cause huge number of open pipes

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. NIFI-6790 Load-balanced queues cause huge number of open pipes

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. NIFI-7200 IPv4 socket resource leak

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          links to

          Web Link GitHub Pull Request #4069

          Activity

            People

              mattyb149 Matt Burgess
              vzolin Vinicius Zolin
              Votes:
              3 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m