Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-8675

Divide Segment Search Amongst Multiple Threads

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • core/search
    • None
    • New

    Description

      Segment search is a single threaded operation today, which can be a bottleneck for large analytical queries which index a lot of data and have complex queries which touch multiple segments (imagine a composite query with range query and filters on top). This ticket is for discussing the idea of splitting a single segment into multiple threads based on mutually exclusive document ID ranges.

      This will be a two phase effort, the first phase targeting queries returning all matching documents (collectors not terminating early). The second phase patch will introduce staged execution and will build on top of this patch.

      Attachments

        1. PhraseHighFreqP50.png
          33 kB
          Atri Sharma
        2. PhraseHighFreqP90.png
          45 kB
          Atri Sharma
        3. TermHighFreqP50.png
          39 kB
          Atri Sharma
        4. TermHighFreqP90.png
          48 kB
          Atri Sharma

        Activity

          People

            Unassigned Unassigned
            atris Atri Sharma
            Votes:
            1 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated: