[SOLR-10525] Stacked recovery requests do no cancel an in progress recovery first.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 7.3, 8.0
• Component/s: SolrCloud
• Assignee: Cao Manh Dat
• Reporter: Mike Drob
• Created: 19/Apr/17 20:10
• Updated: 02/Oct/19 17:23
• Resolved: 29/Jan/18 09:01

Description

https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/update/DefaultSolrCoreState.java#L300-L310

Two issues with this code:

          boolean locked = recoveryLock.tryLock();
          try {
            if (!locked) {
              if (recoveryWaiting.get() > 0) { // line 1
                return;
              }
              recoveryWaiting.incrementAndGet(); // line 2
            } else {
              recoveryWaiting.incrementAndGet();
              cancelRecovery(); // line 3
}

The cancelRecovery on line 3 call will only hit when there are no recoveries to actually cancel (since we got the lock that means there are no recoveries in progress). Instead it should be moved either to the either branch of the if, or outside after the if since we know we will be running a recovery at that point.

This code doesn't always prevent multiple requests from stacking. If there is a recovery running, but no recoveries currently waiting, multiple requests can check the count at line 1 before any of them will increment the count at line 2 and thus all of them will hit the increment.

I don't have specific tests for this, but it's causing failures for me on my SOLR-9555 work in progress.

Issue Links

• is depended upon by: SOLR-9555 Leader incorrectly publishes state for replica when it puts replica into LIR.
• relates to: SOLR-8371 Try and prevent too many recovery requests from stacking up and clean up some faulty logic.