Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-14123

autoAddReplicas is not reliable when multiple nodes go down.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 8.3
    • Fix Version/s: None
    • Component/s: AutoScaling
    • Labels:

      Description

      I started noticing problems in our production environment with indexing being blocked due to a minimum replication factor not being met.  We have autoAddReplicas triggers in place to add replicas when nodes our lost but it doesn't seem to correctly add all replicas that have been lost when nodes are lost. I’ve been able to reproduce this behavior consistently in a development environment.

      Repro:

      1. Setup a 10 node SolrCloud cluster.
      2. Create autoAddReplicas to trigger on nodeLost with waitFor set to 10 minutes.
      3. Create 15 collections with 2 shards and 4 replicas.
      4. Kill 3 Solr nodes.
      5. 15 minutes later kill 1 more Solr node.

      Results:

      Monitor your shards/replicas.  You’ll see some replicas added to make up for the lost replicas but not all.  An hour later many shards are still missing replicas.

      Expected:

      All lost replicas should be added on the 6 remaining healthy nodes.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              dhunt David Hunt
            • Votes:
              2 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated: