Uploaded image for project: 'Jackrabbit Oak'
  1. Jackrabbit Oak
  2. OAK-4412

Lucene hybrid index

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.5.11, 1.6.0
    • Component/s: lucene
    • Labels:
      None

      Description

      When running Oak in a cluster, each write operation is expensive. After performing some stress-tests with a geo-distributed Mongo cluster, we've found out that updating property indexes is a large part of the overall traffic.

      The asynchronous index would be an answer here (as the index update won't be made in the client request thread), but the AEM requires the updates to be visible immediately in order to work properly.

      The idea here is to enhance the existing asynchronous Lucene index with a synchronous, locally-stored counterpart that will persist only the data since the last Lucene background reindexing job.

      The new index can be stored in memory or (if necessary) in MMAPed local files. Once the "main" Lucene index is being updated, the local index will be purged.

      Queries will use an union of results from the lucene and lucene-memory indexes.

      The lucene-memory index, as a local stored entity, will be updated using an observer, so it'll get both local and remote changes.

      The original idea has been suggested by Chetan Mehrotra in the discussion for the OAK-4233.

      Feature Docs
      http://jackrabbit.apache.org/oak/docs/query/indexing.html#nrt-indexing

        Attachments

        1. hybrid-result-v1.txt
          162 kB
          Chetan Mehrotra
        2. hybrid-benchmark.sh
          2 kB
          Chetan Mehrotra
        3. OAK-4412-v1.diff
          196 kB
          Chetan Mehrotra
        4. OAK-4412.patch
          87 kB
          Tomek Rękawek

          Issue Links

            Activity

              People

              • Assignee:
                chetanm Chetan Mehrotra
                Reporter:
                tomek.rekawek Tomek Rękawek
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Due:
                  Created:
                  Updated:
                  Resolved: