[SOLR-14927] Remove Overseer

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: SolrCloud
• Labels: cluster collection-api overseer solrcloud zookeeper
• Assignee: Ilan Ginzburg
• Reporter: Ilan Ginzburg
• Created: 13/Oct/20 10:06
• Updated: 03/Oct/23 01:12

Description

This Jira is intended to capture sub jiras on the path to remove the Overseer component from SolrCloud and move to all nodes being able to do the work currently done by Overseer.

See detailed description in this doc.

Copying (edited) from the above doc:

The motivation for removing Overseer include:

• Mono threaded state change is slow and doesn’t scale,
• Communication between cluster nodes and the Overseer use Zookeeper as a queueing mechanism, this is not a good idea,
• Nodes talking to Overseer (then Overseer talking to itself) via Zookeeper is inefficient and adds latency,
• Collection API scalability is poor, because not only a single node processes commands for all Collections, but it also depends on the mono threaded state change queue consumption,
• The code supporting Overseer in SolrCloud is complex (election, queue management, recovery etc).

The general idea is that there’s already a central point in the SolrCloud cluster and it’s Zookeeper. It might not be necessary to have a second central point (Overseer) because nodes can interact directly with Zookeeper and synchronize more efficiently by optimistic locking using “conditional updates” (a.k.a compare and swap or CAS).

 

Issue Links

• is related to: SOLR-11465 Overseer should process independent messages in parallel

Sub-Tasks

• Remove Overseer ClusterStateUpdater: Closed
• Distribute Collection API command execution: Closed
• Refactor: separate Collection API commands from Overseer and message handling logic: Closed
• Remove OverseerConfigSetMessageHandler; always do distributed mode: Open