[SOLR-11285] Support simulations at scale in the autoscaling framework

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 7.3, 8.0
• Component/s: AutoScaling
• Assignee: Andrzej Bialecki
• Reporter: Andrzej Bialecki
• Created: 24/Aug/17 13:07
• Updated: 08/Jun/19 15:19
• Resolved: 19/Dec/17 12:30

Description

This is a spike to investigate how difficult it would be to modify the autoscaling framework so that it's possible to run simulated large-scale experiments and test its dynamic behavior without actually spinning up a large cluster.
Currently many components rely heavily on actual Solr, ZK and behavior of ZK watches, or insist on making actual HTTP calls. Notable exception is the core Policy framework where most of the ZK / Solr details are abstracted.
As the algorithms for autoscaling that we implement become more and more complex the ability to effectively run multiple large simulations will be crucial - it's very easy to unknowingly introduce catastrophic instabilities that don't manifest themselves in regular unit tests.

Issue Links

• is blocked by: SOLR-11714 AddReplicaSuggester endless loop

Sub-Tasks

• refactor DistibutedQueue to an interface: Resolved
• Implement simulated DistribStateManager: Closed
• Implement simulated ClusterDataProvider: Closed
• Implement simulated DistributedQueue: Closed
• Convert some autoscaling tests to use the simulation framework: Closed
• Implement ObjectCache for keeping shared state in SolrCloudManager: Closed