Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9191

OverseerTaskQueue.peekTopN() fatally flawed

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 5.4, 5.4.1, 5.5, 5.5.1, 6.0, 6.0.1
    • Fix Version/s: 5.5.2, 5.6, 6.0.2, 6.1, 6.2, 7.0
    • Component/s: None
    • Labels:
      None

      Description

      We rewrote DistributedQueue in SOLR-6760, to optimize its obvious use case as a FIFO. But in doing so, we broke the assumptions in OverseerTaskQueue.peekTopN()..

      OverseerTaskQueue.peekTopN() involves filtering out items you're already working on, it's trying to peek for new items in the queue beyond what you already know about. But DistributedQueue (being designed as a FIFO) doesn't know about the filtering; as long as it has any items in-memory it just keeps returning those over and over without ever pulling new data from ZK. This is true even if the watcher has fired and marked the state as dirty. So OverseerTaskQueue gets into a state where it can never read new items in ZK because DQ keeps returning the same items that it has marked as in-progress.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                dragonsinth Scott Blum
                Reporter:
                dragonsinth Scott Blum
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 24h
                  24h
                  Remaining:
                  Remaining Estimate - 24h
                  24h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified