Solr
  1. Solr
  2. SOLR-3559

handle deleteByQuery reorders in distributed indexing

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.0-BETA
    • Component/s: SolrCloud
    • Labels:
      None

      Description

      We currently handle other update reorders (i.e. replica A may see update X then Y, and replica B may see update Y then X). deleteByQuery also needs to handle reorders to ensure replicas are consistent.

      1. SOLR-3559.patch
        28 kB
        Yonik Seeley
      2. SOLR-3559.patch
        17 kB
        Yonik Seeley
      3. SOLR-3559.patch
        15 kB
        Yonik Seeley
      4. SOLR-3559.patch
        12 kB
        Yonik Seeley
      5. SOLR-3559.patch
        7 kB
        Yonik Seeley

        Activity

        Hide
        Yonik Seeley added a comment -

        Here's a draft patch... no tests yet.

        Show
        Yonik Seeley added a comment - Here's a draft patch... no tests yet.
        Hide
        Yonik Seeley added a comment -

        Updated patch with bug fixes and a simple test for reordering that passes. Unfortunately some other tests now fail.

        Show
        Yonik Seeley added a comment - Updated patch with bug fixes and a simple test for reordering that passes. Unfortunately some other tests now fail.
        Hide
        Yonik Seeley added a comment -

        OK, additional bug fixes plus a fix to some of the tests that faked doc versions. clearIndex() with it's DBQ of : was messing things up since it could also delete documents in the future (docs that were using fake versions that were lower than the DBQ).

        Time to add in the stress tests.

        Show
        Yonik Seeley added a comment - OK, additional bug fixes plus a fix to some of the tests that faked doc versions. clearIndex() with it's DBQ of : was messing things up since it could also delete documents in the future (docs that were using fake versions that were lower than the DBQ). Time to add in the stress tests.
        Hide
        Yonik Seeley added a comment -

        Here's the latest patch. Unfortunately stress tests fail. I'm not sure at this point if it's a logic bug, a concurrency bug, or a test bug...

        Show
        Yonik Seeley added a comment - Here's the latest patch. Unfortunately stress tests fail. I'm not sure at this point if it's a logic bug, a concurrency bug, or a test bug...
        Hide
        Yonik Seeley added a comment -

        Narrowing the issues down, the stress tests only fail with greater than 1 write thread in conjunction with realtime-get.

        Show
        Yonik Seeley added a comment - Narrowing the issues down, the stress tests only fail with greater than 1 write thread in conjunction with realtime-get.
        Hide
        Yonik Seeley added a comment -

        Latest version that seems to pass stress tests. I had to handle the add and associated deleteByQueries more as a single unit in the update handler rather than in the distrib update processor.

        Show
        Yonik Seeley added a comment - Latest version that seems to pass stress tests. I had to handle the add and associated deleteByQueries more as a single unit in the update handler rather than in the distrib update processor.
        Hide
        Yonik Seeley added a comment -

        OK, I think I finally nailed the last bug in this. DBQ needs to clear all maps in the update log (i.e. prevMap, prevMap2) otherwise lookups can get a false hit during a concurrent commit.

        Show
        Yonik Seeley added a comment - OK, I think I finally nailed the last bug in this. DBQ needs to clear all maps in the update log (i.e. prevMap, prevMap2) otherwise lookups can get a false hit during a concurrent commit.
        Hide
        Hoss Man added a comment -

        hoss20120711-manual-post-40alpha-change

        Show
        Hoss Man added a comment - hoss20120711-manual-post-40alpha-change
        Hide
        Yonik Seeley added a comment -

        Committed http://svn.apache.org/viewvc?view=revision&revision=1365014
        to trunk and 4x branch that fixes DBQ still doing leader-type stuff even when FROMLEADER was set.

        Show
        Yonik Seeley added a comment - Committed http://svn.apache.org/viewvc?view=revision&revision=1365014 to trunk and 4x branch that fixes DBQ still doing leader-type stuff even when FROMLEADER was set.

          People

          • Assignee:
            Yonik Seeley
            Reporter:
            Yonik Seeley
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development