Uploaded image for project: 'Jackrabbit Oak'
  1. Jackrabbit Oak
  2. OAK-2723

FileStore does not scale because of precomputed graph on TarReader

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 1.1.8
    • 1.2.1, 1.3.0, 1.4
    • core
    • None
    • Patch

    Description

      The FileStore keeps a reference to all TarReader object, one per each file. In my test, for an ~350 Gb repository, that was ~1100 tar files, with a TarReader for each.

      The problem is TarReader keeps a reference to a precomputed graph ByteBuffer, which is not really used that much. That means that through the readers field, there's a reference to these graphs, which means they can't be GC'ed.

      The construction of FileStore is from oak-run:

      FileStore store = new FileStore(directory, 256, TAR_STORAGE_MEMORY_MAPPED);

      The effect is you need more that 6GB of Ram just to instantiate the FileStore object.

      The attached patch fixes this issue.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            mduerig Michael Dürig
            andrei.dulvac Andrei Dulvac
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment