Lucene - Core
  1. Lucene - Core
  2. LUCENE-769

[PATCH] Performance improvement for some cases of sorted search

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: 2.0.0
    • Fix Version/s: None
    • Component/s: core/search
    • Labels:
      None
    • Lucene Fields:
      New, Patch Available

      Description

      It's a small addition to Lucene that significantly lowers memory consumption and improves performance for sorted searches with frequent index updates and relatively big indexes (>1mln docs) scenario. This solution supports only single-field sorting currently (which seem to be quite popular use case). Multiple fields support can be added without much trouble.

      The solution is this: documents from the sorting set (instead of given field's values from the whole index - current FieldCache approach) are cached in a WeakHashMap so the cached items are candidates for GC. Their fields values are then fetched from the cache and compared while sorting.

      1. selfContained.patch
        12 kB
        Artem Vasiliev

        Activity

        Artem Vasiliev created issue -
        Artem Vasiliev made changes -
        Field Original Value New Value
        Attachment DocCachingSorting.patch [ 12348593 ]
        Artem Vasiliev made changes -
        Attachment DocCachingSorting.patch [ 12348678 ]
        Artem Vasiliev made changes -
        Attachment StoredFieldSorting.patch [ 12348757 ]
        robert engels made changes -
        Attachment QueryFilter.java [ 12348767 ]
        robert engels made changes -
        Attachment IndexReaderUtils.java [ 12348771 ]
        Artem Vasiliev made changes -
        Attachment selfContained.patch [ 12348860 ]
        Artem Vasiliev made changes -
        Attachment selfContained.patch [ 12348861 ]
        Artem Vasiliev made changes -
        Attachment selfContained.patch [ 12348862 ]
        Artem Vasiliev made changes -
        Attachment selfContained.patch [ 12350080 ]
        Artem Vasiliev made changes -
        Description It's a small addition to Lucene that significantly lowers memory consumption and improves performance for sorted searches with frequent index updates and relatively big indexes (>1mln docs) scenario. This solution supports only single-field sorting currently (which seem to be quite popular use case). Multiple fields support can be added without much trouble.

        The solution is this: documents from the sorting set (instead of given field's values from the whole index - current FieldCache approach) are cached in a WeakHashMap so the cached items are candidates for GC. Their fields values are then fetched from the cache and compared while sorting.
        The attachment №5 from 31 Jan is the last and the whole patch for this issue. Guys with permissions, please remove all the others.

        It's a small addition to Lucene that significantly lowers memory consumption and improves performance for sorted searches with frequent index updates and relatively big indexes (>1mln docs) scenario. This solution supports only single-field sorting currently (which seem to be quite popular use case). Multiple fields support can be added without much trouble.

        The solution is this: documents from the sorting set (instead of given field's values from the whole index - current FieldCache approach) are cached in a WeakHashMap so the cached items are candidates for GC. Their fields values are then fetched from the cache and compared while sorting.
        Karl Wettin made changes -
        Attachment selfContained.patch [ 12348862 ]
        Karl Wettin made changes -
        Attachment selfContained.patch [ 12348861 ]
        Karl Wettin made changes -
        Attachment selfContained.patch [ 12348860 ]
        Karl Wettin made changes -
        Attachment StoredFieldSorting.patch [ 12348757 ]
        Karl Wettin made changes -
        Attachment DocCachingSorting.patch [ 12348678 ]
        Karl Wettin made changes -
        Attachment DocCachingSorting.patch [ 12348593 ]
        Karl Wettin made changes -
        Attachment IndexReaderUtils.java [ 12348771 ]
        Karl Wettin made changes -
        Attachment QueryFilter.java [ 12348767 ]
        Artem Vasiliev made changes -
        Description The attachment №5 from 31 Jan is the last and the whole patch for this issue. Guys with permissions, please remove all the others.

        It's a small addition to Lucene that significantly lowers memory consumption and improves performance for sorted searches with frequent index updates and relatively big indexes (>1mln docs) scenario. This solution supports only single-field sorting currently (which seem to be quite popular use case). Multiple fields support can be added without much trouble.

        The solution is this: documents from the sorting set (instead of given field's values from the whole index - current FieldCache approach) are cached in a WeakHashMap so the cached items are candidates for GC. Their fields values are then fetched from the cache and compared while sorting.
        It's a small addition to Lucene that significantly lowers memory consumption and improves performance for sorted searches with frequent index updates and relatively big indexes (>1mln docs) scenario. This solution supports only single-field sorting currently (which seem to be quite popular use case). Multiple fields support can be added without much trouble.

        The solution is this: documents from the sorting set (instead of given field's values from the whole index - current FieldCache approach) are cached in a WeakHashMap so the cached items are candidates for GC. Their fields values are then fetched from the cache and compared while sorting.
        Robert Muir made changes -
        Component/s Search [ 12310235 ]
        Mark Thomas made changes -
        Workflow jira [ 12393903 ] Default workflow, editable Closed status [ 12563549 ]
        Mark Thomas made changes -
        Workflow Default workflow, editable Closed status [ 12563549 ] jira [ 12585141 ]
        Erick Erickson made changes -
        Status Open [ 1 ] Closed [ 6 ]
        Resolution Won't Fix [ 2 ]

          People

          • Assignee:
            Unassigned
            Reporter:
            Artem Vasiliev
          • Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development