[SOLR-11475] Endless loop and OOM in PeerSync

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Andrey Kudryavtsev
• Created: 12/Oct/17 10:36
• Updated: 02/Oct/19 17:24

Description

After problem described in SOLR-11459, I restarted cluster and got OOM on start.

PeerSync#handleVersionsWithRanges contains this logic:

    while (otherUpdatesIndex >= 0) {
      // we have run out of ourUpdates, pick up all the remaining versions from the other versions
      if (ourUpdatesIndex < 0) {
        String range = otherVersions.get(otherUpdatesIndex) + "..." + otherVersions.get(0);
        rangesToRequest.add(range);
        totalRequestedVersions += otherUpdatesIndex + 1;
        break;
      }

      // stop when the entries get old enough that reorders may lead us to see updates we don't need
      if (!completeList && Math.abs(otherVersions.get(otherUpdatesIndex)) < ourLowThreshold) break;

      if (ourUpdates.get(ourUpdatesIndex).longValue() == otherVersions.get(otherUpdatesIndex).longValue()) {
        ourUpdatesIndex--;
        otherUpdatesIndex--;
      } else if (Math.abs(ourUpdates.get(ourUpdatesIndex)) < Math.abs(otherVersions.get(otherUpdatesIndex))) {
        ourUpdatesIndex--;
      } else {
        long rangeStart = otherVersions.get(otherUpdatesIndex);
        while ((otherUpdatesIndex < otherVersions.size())
            && (Math.abs(otherVersions.get(otherUpdatesIndex)) < Math.abs(ourUpdates.get(ourUpdatesIndex)))) {
          otherUpdatesIndex--;
          totalRequestedVersions++;
        }
        // construct range here
        rangesToRequest.add(rangeStart + "..." + otherVersions.get(otherUpdatesIndex + 1));
      }
    }

If at some point there will be

 ourUpdates.get(ourUpdatesIndex) = -otherVersions.get(otherUpdatesIndex) 

loop will never end. It will add same string again and again into rangesToRequest until process runs out of memory.

Issue Links

• is duplicated by: SOLR-11474 Endless loop and OOM in PeerSync
• is related to: SOLR-9207 PeerSync recovery fails if number of updates requested is high
• is related to: SOLR-11459 AddUpdateCommand#prevVersion is not cleared which may lead to problem for in-place updates of non existed documents