Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-14347

Autoscaling placement wrong when concurrent replica placements are calculated

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 8.5
    • Fix Version/s: 8.6
    • Component/s: AutoScaling
    • Labels:
      None

      Description

      Steps to reproduce:

      • create a cluster of a few nodes (tested with 7 nodes)
      • define per-collection policies that distribute replicas exclusively on different nodes per policy
      • concurrently create a few collections, each using a different policy
      • resulting replica placement will be seriously wrong, causing many policy violations

      Running the same scenario but instead creating collections sequentially results in no violations.

      I suspect this is caused by incorrect locking level for all collection operations (as defined in CollectionParams.CollectionAction) that create new replica placements - i.e. CREATE, ADDREPLICA, MOVEREPLICA, DELETENODE, REPLACENODE, SPLITSHARD, RESTORE, REINDEXCOLLECTION. All of these operations use the policy engine to create new replica placements, and as a result they change the cluster state. However, currently these operations are locked (in OverseerCollectionMessageHandler.lockTask ) using LockLevel.COLLECTION. In practice this means that the lock is held only for the particular collection that is being modified.

      A straightforward fix for this issue is to change the locking level to CLUSTER (and I confirm this fixes the scenario described above). However, this effectively serializes all collection operations listed above, which will result in general slow-down of all collection operations.

        Attachments

        1. SOLR-14347.patch
          2 kB
          Andrzej Bialecki

          Issue Links

            Activity

              People

              • Assignee:
                ab Andrzej Bialecki
                Reporter:
                ab Andrzej Bialecki
              • Votes:
                1 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 20m
                  1h 20m