Uploaded image for project: 'Jackrabbit Oak'
  1. Jackrabbit Oak
  2. OAK-2723

FileStore does not scale because of precomputed graph on TarReader

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 1.1.8
    • 1.2.1, 1.3.0, 1.4
    • core
    • None
    • Patch

    Description

      The FileStore keeps a reference to all TarReader object, one per each file. In my test, for an ~350 Gb repository, that was ~1100 tar files, with a TarReader for each.

      The problem is TarReader keeps a reference to a precomputed graph ByteBuffer, which is not really used that much. That means that through the readers field, there's a reference to these graphs, which means they can't be GC'ed.

      The construction of FileStore is from oak-run:

      FileStore store = new FileStore(directory, 256, TAR_STORAGE_MEMORY_MAPPED);

      The effect is you need more that 6GB of Ram just to instantiate the FileStore object.

      The attached patch fixes this issue.

      Attachments

        Issue Links

          Activity

            People

              mduerig Michael Dürig
              andrei.dulvac Andrei Dulvac
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: