[SOLR-17102] VersionBucket not needed

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 9.8
• Component/s: SolrCloud
• Labels: pull-request-available
• Assignee: David Smiley
• Reporter: David Smiley
• Created: 11/Dec/23 06:00
• Updated: 10/Oct/24 16:55
• Resolved: 02/Sep/24 19:53

Description

SolrCloud ensures that updates for the same document ID are done in the correct order internally in the face of possible re-orders during replication / log replay.  In order to ensure the updates are applied consecutively, a lock is held on a hash of the ID for the doc.  A hash is used to limit the number of total locks because the locks are pre-created in advance for the core (numVersionBuckets == 65k by default).  The memory is non-negligible with many cores, and it introduces the possibility of collisions, especially at lower bucket counts if you configure it much lower.

Here I propose doing away with a pre-created hashed bucket strategy.  Instead, I propose more simply creating and GC'ing a lock per update being processed, and using a ConcurrentHashMap to hold those in-flight.  This strategy is already used in org.apache.solr.util.OrderedExecutor.SparseStripedLock, more or less.

Doing this is more tractable now that VersionBucket only holds a lock, not a version anymore – SOLR-17036

The biggest challenge is that the code calls for the ability to use a Condition to away/notify, which means the solution can't just re-use SparseStripedLock above nor be quite so simple.

Issue Links

• requires: SOLR-17036 Lazy create VersionBucket in the update log to free memory.
• links to: GitHub Pull Request #2548