Uploaded image for project: 'Jackrabbit Oak'
  1. Jackrabbit Oak
  2. OAK-4412

Lucene hybrid index

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 1.5.11, 1.6.0
    • lucene
    • None

    Description

      When running Oak in a cluster, each write operation is expensive. After performing some stress-tests with a geo-distributed Mongo cluster, we've found out that updating property indexes is a large part of the overall traffic.

      The asynchronous index would be an answer here (as the index update won't be made in the client request thread), but the AEM requires the updates to be visible immediately in order to work properly.

      The idea here is to enhance the existing asynchronous Lucene index with a synchronous, locally-stored counterpart that will persist only the data since the last Lucene background reindexing job.

      The new index can be stored in memory or (if necessary) in MMAPed local files. Once the "main" Lucene index is being updated, the local index will be purged.

      Queries will use an union of results from the lucene and lucene-memory indexes.

      The lucene-memory index, as a local stored entity, will be updated using an observer, so it'll get both local and remote changes.

      The original idea has been suggested by chetanm in the discussion for the OAK-4233.

      Feature Docs
      http://jackrabbit.apache.org/oak/docs/query/indexing.html#nrt-indexing

      Attachments

        1. hybrid-benchmark.sh
          2 kB
          Chetan Mehrotra
        2. hybrid-result-v1.txt
          162 kB
          Chetan Mehrotra
        3. OAK-4412.patch
          87 kB
          Tomek Rękawek
        4. OAK-4412-v1.diff
          196 kB
          Chetan Mehrotra

        Issue Links

          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. OAK-4714 Reindexing should only happen in async mode for a mix mode index definition

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. OAK-4566 Multiplexing store support in Lucene Indexes

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. OAK-4640 Provide a way for commit hook to record meta data for a given commit

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. OAK-4641 Using same index definition for both async and sync indexing

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. OAK-4753 Add nrt as another index mode

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. OAK-4909 NRTIndex can get closed while in use

          • Major - Major loss of function.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. OAK-6535 Synchronous Lucene Property Indexes

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. OAK-4904 For unique indexes avoid consulting indexes other than property index

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. OAK-5421 Add LuceneDoc directly to queue from LuceneIndexEditor

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. OAK-4809 JMX Stats for NRT Indexing

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. OAK-4906 Lucene: Support relative property based query by transforming the path

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. OAK-4808 Index external changes as part of NRT indexing

          • Major - Major loss of function.
          • Closed
          requires

          Improvement - An improvement or enhancement to an existing feature or task. OAK-4767 Provide a way to report additional stats as part of benchmark run

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. OAK-4768 Provide an option to enable Metrics collection for benchmark run

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          supercedes

          Improvement - An improvement or enhancement to an existing feature or task. OAK-4233 Property index stored locally

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              chetanm Chetan Mehrotra
              tomek.rekawek Tomek Rękawek
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: