Description
Currently the DQ works as follows
- read all items in the directory
- sort them all
- take the head and return it and discard everything else
- rinse and repeat
This works well when we have only a handful of items in the Queue. If the items in the queue is much larger (in tens of thousands) , this is counterproductive
As the overseer queue is a multiple producers + single consumer queue, We can read them all in bulk and before processing each item , just do a zk.exists(itemname) and if all is well we don't need to do the fetch all + sort thing again
Attachments
Attachments
Issue Links
- relates to
-
SOLR-7191 Improve stability and startup performance of SolrCloud with thousands of collections
-
- Resolved
-