Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-6137

Managed Schema / Schemaless and SolrCloud concurrency issues

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      This is a follow up to a message on the mailing list, linked here: http://mail-archives.apache.org/mod_mbox/lucene-dev/201406.mbox/%3CCAKfebOOcMeVEb010SsdcH8nta%3DyonMK5R7dSFOsbJ_tnre0O7w%40mail.gmail.com%3E

      The Managed Schema integration with SolrCloud seems pretty limited.

      The issue I'm running into is variants of the issue that schema changes are not pushed to all shards/replicas synchronously. So, for example, I can make the following two requests:
      1) add a field to the collection on server1 using the Schema API
      2) add a document with the new field, the document is routed to a core on server2

      Then, there appears to be a race between when the document is processed by the core on server2 and when the core on server2, via the ZkIndexSchemaReader, gets the new schema. If the document is processed first, I get a 400 error because the field doesn't exist. This is easily reproducible by adding a sleep to the ZkIndexSchemaReader's processing.

      I hit a similar issue with Schemaless: the distributed request handler sends out the document updates, but there is no guarantee that the other shards/replicas see the schema changes made by the update.chain.

      Another issue I noticed today: making multiple schema API calls concurrently can block; that is, one may get through and the other may infinite loop.

      So, for reference, the issues include:
      1) Schema API changes return success before all cores are updated; subsequent calls attempting to use new schema may fail
      2) Schemaless changes may fail on replicas/other shards for the same reason
      3) Concurrent Schema API changes may block

      From Steve Rowe on the mailing list:

      For Schema API users, delaying a couple of seconds after adding fields before using them should workaround this problem. While not ideal, I think schema field additions are rare enough in the Solr collection lifecycle that this is not a huge problem.

      For schemaless users, the picture is worse, as you noted. Immediate distribution of documents triggering schema field addition could easily prove problematic. Maybe we need a schema update blocking mode, where after the ZK schema node watch is triggered, all new request processing is halted until the schema is finished downloading/parsing/swapping out? (Such a mode should help Schema API users too.)

      Attachments

        1. SOLR-6137v4.patch
          25 kB
          Gregory Chanan
        2. SOLR-6137v3.patch
          5 kB
          Gregory Chanan
        3. SOLR-6137v2.patch
          25 kB
          Gregory Chanan
        4. SOLR-6137.patch
          27 kB
          Gregory Chanan
        5. SOLR-6137.patch
          28 kB
          Steven Rowe

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            sarowe Steven Rowe
            gchanan Gregory Chanan
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment