Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-11475

Endless loop and OOM in PeerSync

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      After problem described in SOLR-11459, I restarted cluster and got OOM on start.

      PeerSync#handleVersionsWithRanges contains this logic:

          while (otherUpdatesIndex >= 0) {
            // we have run out of ourUpdates, pick up all the remaining versions from the other versions
            if (ourUpdatesIndex < 0) {
              String range = otherVersions.get(otherUpdatesIndex) + "..." + otherVersions.get(0);
              rangesToRequest.add(range);
              totalRequestedVersions += otherUpdatesIndex + 1;
              break;
            }
      
            // stop when the entries get old enough that reorders may lead us to see updates we don't need
            if (!completeList && Math.abs(otherVersions.get(otherUpdatesIndex)) < ourLowThreshold) break;
      
            if (ourUpdates.get(ourUpdatesIndex).longValue() == otherVersions.get(otherUpdatesIndex).longValue()) {
              ourUpdatesIndex--;
              otherUpdatesIndex--;
            } else if (Math.abs(ourUpdates.get(ourUpdatesIndex)) < Math.abs(otherVersions.get(otherUpdatesIndex))) {
              ourUpdatesIndex--;
            } else {
              long rangeStart = otherVersions.get(otherUpdatesIndex);
              while ((otherUpdatesIndex < otherVersions.size())
                  && (Math.abs(otherVersions.get(otherUpdatesIndex)) < Math.abs(ourUpdates.get(ourUpdatesIndex)))) {
                otherUpdatesIndex--;
                totalRequestedVersions++;
              }
              // construct range here
              rangesToRequest.add(rangeStart + "..." + otherVersions.get(otherUpdatesIndex + 1));
            }
          }
      

      If at some point there will be

       ourUpdates.get(ourUpdatesIndex) = -otherVersions.get(otherUpdatesIndex) 

      loop will never end. It will add same string again and again into rangesToRequest until process runs out of memory.

        Attachments

        1. SOLR-11475.patch
          4 kB
          Pushkar Raste

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                werder Andrey Kudryavtsev
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: