Solr
  1. Solr
  2. SOLR-7625

Version bucket seed not updated after new index is installed on a replica

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 5.2
    • Fix Version/s: 5.2, 6.0
    • Component/s: SolrCloud
    • Labels:
      None
    • Flags:
      Important

      Description

      This is related to SOLR-7332 ... I used to set the highest value on the version buckets based on a lookup to the index whenever a firstSearcher event fired, but that led to reentrant lock issues (see SOLR-7587), so I moved that version seeding code into the SolrCore constructor.

      While working on SOLR-4506, I just realized that if a core has to pull index files from the leader to recover, then the version bucket doesn't get updated once the new index files are pulled over. In other words, the code seeds the version buckets during core init, but if the underlying index changes due to being too far out-of-date, then the version buckets are never updated once the new index is in place. The risk is that if bucket versions are seeded with a value that is too low, then it can lead to correctness issues with versioned updates.

      1. SOLR-7625.patch
        7 kB
        Timothy Potter

        Issue Links

          Activity

          Hide
          Anshum Gupta added a comment -

          Good we found this just in time! I'll hold on and also fix other minor issues.

          Show
          Anshum Gupta added a comment - Good we found this just in time! I'll hold on and also fix other minor issues.
          Hide
          Timothy Potter added a comment -

          Here's a patch that addresses this issue. The key part of the fix is in RecoveryStrategy where it will call core.seedVersionBuckets() after recovery is successful if the tlog wasn't replayed; if the tlog was replayed, then the version buckets are already updated as part of the replay process. So this patch catches the problem where no replay happens, but the underlying index still gets updated (usually by downloading a new index dir from the leader).

          For now, I had to infuse a test for this into the HttpPartitionTest because I need a replica to go into recovery to verify that that version bucket max is updated after recovery succeeds.

          All tests pass on trunk with this patch applied.

          Show
          Timothy Potter added a comment - Here's a patch that addresses this issue. The key part of the fix is in RecoveryStrategy where it will call core.seedVersionBuckets() after recovery is successful if the tlog wasn't replayed; if the tlog was replayed, then the version buckets are already updated as part of the replay process. So this patch catches the problem where no replay happens, but the underlying index still gets updated (usually by downloading a new index dir from the leader). For now, I had to infuse a test for this into the HttpPartitionTest because I need a replica to go into recovery to verify that that version bucket max is updated after recovery succeeds. All tests pass on trunk with this patch applied.
          Hide
          Anshum Gupta added a comment -

          Did a quick review. LGTM. +1.

          Show
          Anshum Gupta added a comment - Did a quick review. LGTM. +1.
          Hide
          ASF subversion and git services added a comment -

          Commit 1683174 from Timothy Potter in branch 'dev/trunk'
          [ https://svn.apache.org/r1683174 ]

          SOLR-7625: Ensure that the max value for seeding version buckets is updated after recovery even if the UpdateLog is not replayed.

          Show
          ASF subversion and git services added a comment - Commit 1683174 from Timothy Potter in branch 'dev/trunk' [ https://svn.apache.org/r1683174 ] SOLR-7625 : Ensure that the max value for seeding version buckets is updated after recovery even if the UpdateLog is not replayed.
          Hide
          ASF subversion and git services added a comment -

          Commit 1683175 from Timothy Potter in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1683175 ]

          SOLR-7625: Ensure that the max value for seeding version buckets is updated after recovery even if the UpdateLog is not replayed.

          Show
          ASF subversion and git services added a comment - Commit 1683175 from Timothy Potter in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1683175 ] SOLR-7625 : Ensure that the max value for seeding version buckets is updated after recovery even if the UpdateLog is not replayed.
          Hide
          ASF subversion and git services added a comment -

          Commit 1683177 from Timothy Potter in branch 'dev/branches/lucene_solr_5_2'
          [ https://svn.apache.org/r1683177 ]

          SOLR-7625: Ensure that the max value for seeding version buckets is updated after recovery even if the UpdateLog is not replayed.

          Show
          ASF subversion and git services added a comment - Commit 1683177 from Timothy Potter in branch 'dev/branches/lucene_solr_5_2' [ https://svn.apache.org/r1683177 ] SOLR-7625 : Ensure that the max value for seeding version buckets is updated after recovery even if the UpdateLog is not replayed.
          Hide
          Anshum Gupta added a comment -

          Bulk close for 5.2.0.

          Show
          Anshum Gupta added a comment - Bulk close for 5.2.0.

            People

            • Assignee:
              Timothy Potter
              Reporter:
              Timothy Potter
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development