Uploaded image for project: 'Jackrabbit Oak'
  1. Jackrabbit Oak
  2. OAK-5192

Reduce Lucene related growth of repository size

    XMLWordPrintableJSON

Details

    Description

      I observed Lucene indexing contributing to up to 99% of repository growth. While the size of the index itself is well inside reasonable bounds, the overall turnover of data being written and removed again can be as much as 99%.

      In the case of the TarMK this negatively impacts overall system performance due to fast growing number of tar files / segments, bad locality of reference, cache misses/thrashing when looking up segments and vastly prolonged garbage collection cycles.

      Attachments

        1. Screen Shot 2017-07-03 at 16.50.00.png
          107 kB
          Tommaso Teofili
        2. OAK-5192.0.patch
          10 kB
          Tommaso Teofili
        3. nonBinSizeTotal.txt
          6 kB
          Michael Dürig
        4. diff.txt.zip
          2.42 MB
          Michael Dürig
        5. binSizeTotal.txt
          8 kB
          Michael Dürig
        6. binSize16384.txt
          7 kB
          Michael Dürig
        7. binSize100.txt
          6 kB
          Michael Dürig
        8. added-bytes-zoom.png
          184 kB
          Michael Dürig

        Issue Links

          breaks

          Bug - A problem which impairs or prevents the functions of the product. OAK-6704 Set default merge polity to tiered as CommitMitigatingTieredMergePolicy seems to be bad for performance

          • Major - Major loss of function.
          • Closed
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. OAK-6269 Support non chunk storage in OakDirectory

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. OAK-6710 CommitMitigated merge policy should not reduce performance significantly

          • Major - Major loss of function.
          • Closed
          relates to

          New Feature - A new feature of the product, which has yet to be developed. OAK-6514 Make Lucene merge policy configurable

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. OAK-6211 Optimize indexing for file-system-centric deployments

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. OAK-6412 Consider upgrading to newer Lucene versions

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. OAK-2808 Active deletion of 'deleted' Lucene index files from DataStore without relying on full scale Blob GC

          • Major - Major loss of function.
          • Closed

          Activity

            People

              teofili Tommaso Teofili
              mduerig Michael Dürig
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: