Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9735 Umbrella JIRA for Auto Scaling and Cluster Management in SolrCloud
  3. SOLR-11320

Lock autoscaling triggers when changes they requested are being made

Attach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments


    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 7.2, 8.0
    • AutoScaling
    • None


      Autoscaling triggers generate events that are then processed by actions such as ComputePlanAction and ExecutePlanAction. This process is far from instantaneous - it may take sometimes several seconds or even minutes to eg. move or add replicas.

      The original condition that caused the first event will usually persist during this time, and eventually after waitFor time elapsed it will lead to a new event being generated, which will be queued for execution once the previous actions are completed - but by that time the original condition may have been alleviated by these actions, and the conditions reported in the new event no longer reflect the latest cluster state.

      For this reason some autoscaling frameworks introduce a "cooldown" period, where triggers are temporarily disabled for a fixed period of time to avoid piling up new events while cluster changes are being made. This method introduces a fixed delay that is specific to a trigger.

      From the point of view of control theory the feedback loop design should minimize inherent delays because they are very hard to properly compensate for and either lead to instability (when controller tries to compensate for an out-of-step state) or lead to increased system lag (the system sluggishly reacts to changes because it has to wait for things to settle down) - so from this point of view a fixed delay, which is also hard to estimate properly and may be inadequate depending on varying conditions, is not ideal.

      A better alternative would be to lock the trigger just for the actual duration of time while changes are being made. Initially this could be implemented as a global lock for all triggers for the duration of modifications performed by ExecutePlanAction.

      Currently cluster modifications executed by ExecutePlanAction are made asynchronously, so it's hard to determine when the changes actually take effect, eg. when a new (or moved) replica becomes active, so this would have to be changed as well.


        Issue Links


          This comment will be Viewable by All Users Viewable by All Users


            ab Andrzej Bialecki
            ab Andrzej Bialecki
            0 Vote for this issue
            2 Start watching this issue




                Issue deployment