[SOLR-15052] Reducing overseer bottlenecks using per-replica states

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 8.8
• Assignee: Noble Paul
• Reporter: Ishan Chattopadhyaya
• Created: 16/Dec/20 10:04
• Updated: 24/Feb/21 05:42
• Resolved: 18/Jan/21 10:12

Description

This work has the same goal as SOLR-13951, that is to reduce overseer bottlenecks by avoiding replica state updates from going to the state.json via the overseer. However, the approach taken here is different from SOLR-13951 and hence this work supercedes that work.

The design proposed is here: https://docs.google.com/document/d/1xdxpzUNmTZbk0vTMZqfen9R3ArdHokLITdiISBxCFUg/edit

Briefly,

1. Every replica's state will be in a separate znode nested under the state.json. It has the name that encodes the replica name, state, leadership status.
2. An additional children watcher to be set on state.json for state changes.
3. Upon a state change, a ZK multi-op to delete the previous znode and add a new znode with new state.

Differences between this and SOLR-13951,

1. In SOLR-13951, we planned to leverage shard terms for per shard states.
2. As a consequence, the code changes required for SOLR-13951 were massive (we needed a shard state provider abstraction and introduce it everywhere in the codebase).
3. This approach is a drastically simpler change and design.

Credits for this design and the PR is due to noble.paul. [e-mail], noble.paul and I have collaborated on this effort. The reference branch takes a conceptually similar (but not identical) approach.

I shall attach a PR and performance benchmarks shortly.

Issue Links

• supercedes: SOLR-13951 Avoid replica state updates to state.json
• links to: GitHub Pull Request #2148
• links to: GitHub Pull Request #2177
• links to: GitHub Pull Request #2187