Uploaded image for project: 'Apache Jena'
  1. Apache Jena
  2. JENA-1615

Compaction leaks file descriptors

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Jena 3.8.0
    • Fix Version/s: Jena 3.10.0
    • Component/s: Core, TDB2
    • Labels:
      None
    • Environment:

      Description

      Context

      I'm using a TDB2 dataset in a long-running Scala application, in which the dataset gets compacted regularly. After compactions, the application removes the Data-xxxx folder of the previous generation. However, the corresponding disk space isn't properly returned back to the OS, but is still reported as being used by df. Indeed, lsof shows that the application keeps open file descriptors that point to the old generation's files. Only stopping / restarting the JVM frees the disk space for good.

      Reproduction steps

      • Connect to an existing TDB2 dataset
        val dataset = TDB2Factory.connectDataset("sample")

      The last sceenshot shows that, even after garbage collection, there are still open file descriptors pointing to the old generation Data-0001.

      Impact

      Depending on how disk usage is being reported, this can be quite problematic. In our case, we're running on an OpenShift infrastructure with limited storage. After only a handful of compactions, the storage is considered full and cannot be used anymore.

        Attachments

        1. open_files_after_compaction_after_gc_with_fix.png
          435 kB
          Damien Obrist
        2. open_files_after_compaction_after_gc.png
          435 kB
          Damien Obrist
        3. open_files_after_compaction_before_gc.png
          434 kB
          Damien Obrist
        4. open_files_before_compaction.png
          268 kB
          Damien Obrist

          Issue Links

            Activity

              People

              • Assignee:
                andy Andy Seaborne
                Reporter:
                dobrist Damien Obrist
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: