Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-6760

New optimized DistributedQueue implementation for overseer

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 5.4, 6.0
    • SolrCloud
    • None

    Description

      Currently the DQ works as follows

      • read all items in the directory
      • sort them all
      • take the head and return it and discard everything else
      • rinse and repeat

      This works well when we have only a handful of items in the Queue. If the items in the queue is much larger (in tens of thousands) , this is counterproductive

      As the overseer queue is a multiple producers + single consumer queue, We can read them all in bulk and before processing each item , just do a zk.exists(itemname) and if all is well we don't need to do the fetch all + sort thing again

      Attachments

        1. deadlock.patch
          56 kB
          Scott Blum
        2. SOLR-6760.patch
          58 kB
          Scott Blum
        3. SOLR-6760.patch
          58 kB
          Scott Blum
        4. SOLR-6760.patch
          57 kB
          Scott Blum
        5. SOLR-6760.patch
          57 kB
          Scott Blum
        6. SOLR-6760-branch_5x.patch
          59 kB
          Shalin Shekhar Mangar

        Issue Links

          Activity

            People

              shalin Shalin Shekhar Mangar
              noble.paul Noble Paul
              Votes:
              4 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: