Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-6810

Faster searching limited but high rows across many shards all with many hits

    XMLWordPrintableJSON

Details

    Description

      Searching "limited but high rows across many shards all with many hits" is slow
      E.g.

      • Query from outside client: q=something&rows=1000
      • Resulting in sub-requests to each shard something a-la this
        • 1) q=something&rows=1000&fl=id,score
        • 2) Request the full documents with ids in the global-top-1000 found among the top-1000 from each shard

      What does the subject mean

      • "limited but high rows" means 1000 in the example above
      • "many shards" means 200-1000 in our case
      • "all with many hits" means that each of the shards have a significant number of hits on the query
        The problem grows on all three factors above

      Doing such a query on our system takes between 5 min to 1 hour - depending on a lot of things. It ought to be much faster, so lets make it.

      Profiling show that the problem is that it takes lots of time to access the store to get id’s for (up to) 1000 docs (value of rows parameter) per shard. Having 1000 shards its up to 1 mio ids that has to be fetched. There is really no good reason to ever read information from store for more than the overall top-1000 documents, that has to be returned to the client.

      For further detail see mail-thread "Slow searching limited but high rows across many shards all with high hits" started 13/11-2014 on dev@lucene.apache.org

      Attachments

        1. branch_5x_rev1642874.patch
          91 kB
          Per Steffensen
        2. branch_5x_rev1642874.patch
          89 kB
          Per Steffensen
        3. branch_5x_rev1645549.patch
          91 kB
          Per Steffensen
        4. SOLR-6810-hack-eoe.patch
          3 kB
          Erick Erickson
        5. SOLR-6810-trunk.patch
          104 kB
          Shalin Shekhar Mangar
        6. SOLR-6810-trunk.patch
          102 kB
          Shalin Shekhar Mangar
        7. SOLR-6810-trunk.patch
          94 kB
          Shalin Shekhar Mangar

        Issue Links

          Activity

            People

              shalin Shalin Shekhar Mangar
              steff1193 Per Steffensen
              Votes:
              1 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated: