Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-5681

Make the OverseerCollectionProcessor multi-threaded

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.9, 6.0
    • Component/s: SolrCloud
    • Labels:
      None

      Description

      Right now, the OverseerCollectionProcessor is single threaded i.e submitting anything long running would have it block processing of other mutually exclusive tasks.
      When OCP tasks become optionally async (SOLR-5477), it'd be good to have truly non-blocking behavior by multi-threading the OCP itself.

      For example, a ShardSplit call on Collection1 would block the thread and thereby, not processing a create collection task (which would stay queued in zk) though both the tasks are mutually exclusive.

      Here are a few of the challenges:

      • Mutual exclusivity: Only let mutually exclusive tasks run in parallel. An easy way to handle that is to only let 1 task per collection run at a time.
      • ZK Distributed Queue to feed tasks: The OCP consumes tasks from a queue. The task from the workQueue is only removed on completion so that in case of a failure, the new Overseer can re-consume the same task and retry. A queue is not the right data structure in the first place to look ahead i.e. get the 2nd task from the queue when the 1st one is in process. Also, deleting tasks which are not at the head of a queue is not really an 'intuitive' thing.

      Proposed solutions for task management:

      • Task funnel and peekAfter(): The parent thread is responsible for getting and passing the request to a new thread (or one from the pool). The parent method uses a peekAfter(last element) instead of a peek(). The peekAfter returns the task after the 'last element'. Maintain this request information and use it for deleting/cleaning up the workQueue.
      • Another (almost duplicate) queue: While offering tasks to workQueue, also offer them to a new queue (call it volatileWorkQueue?). The difference is, as soon as a task from this is picked up for processing by the thread, it's removed from the queue. At the end, the cleanup is done from the workQueue.

        Attachments

        1. SOLR-5681-2.patch
          79 kB
          Anshum Gupta
        2. SOLR-5681-2.patch
          64 kB
          Anshum Gupta
        3. SOLR-5681-2.patch
          66 kB
          Anshum Gupta
        4. SOLR-5681-2.patch
          66 kB
          Anshum Gupta
        5. SOLR-5681-2.patch
          66 kB
          Anshum Gupta
        6. SOLR-5681-2.patch
          66 kB
          Anshum Gupta
        7. SOLR-5681-2.patch
          69 kB
          Noble Paul
        8. SOLR-5681-2.patch
          68 kB
          Noble Paul
        9. SOLR-5681-2.patch
          69 kB
          Anshum Gupta
        10. SOLR-5681-2.patch
          73 kB
          Anshum Gupta
        11. SOLR-5681-2.patch
          73 kB
          Anshum Gupta
        12. SOLR-5681-2.patch
          75 kB
          Anshum Gupta
        13. SOLR-5681-2.patch
          75 kB
          Anshum Gupta
        14. SOLR-5681-2.patch
          75 kB
          Anshum Gupta
        15. SOLR-5681.patch
          20 kB
          Anshum Gupta
        16. SOLR-5681.patch
          23 kB
          Anshum Gupta
        17. SOLR-5681.patch
          30 kB
          Anshum Gupta
        18. SOLR-5681.patch
          30 kB
          Noble Paul
        19. SOLR-5681.patch
          35 kB
          Anshum Gupta
        20. SOLR-5681.patch
          37 kB
          Anshum Gupta
        21. SOLR-5681.patch
          35 kB
          Anshum Gupta
        22. SOLR-5681.patch
          35 kB
          Anshum Gupta
        23. SOLR-5681.patch
          36 kB
          Anshum Gupta
        24. SOLR-5681.patch
          37 kB
          Anshum Gupta
        25. SOLR-5681.patch
          36 kB
          Anshum Gupta
        26. SOLR-5681.patch
          36 kB
          Anshum Gupta
        27. SOLR-5681.patch
          40 kB
          Anshum Gupta
        28. SOLR-5681.patch
          46 kB
          Anshum Gupta
        29. SOLR-5681.patch
          45 kB
          Anshum Gupta
        30. SOLR-5681.patch
          44 kB
          Anshum Gupta
        31. SOLR-5681.patch
          44 kB
          Anshum Gupta
        32. SOLR-5681.patch
          44 kB
          Anshum Gupta
        33. SOLR-5681.patch
          45 kB
          Anshum Gupta

          Issue Links

            Activity

              People

              • Assignee:
                anshumg Anshum Gupta
                Reporter:
                anshumg Anshum Gupta
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: