Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9835

Create another replication mode for SolrCloud

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 7.0
    • Component/s: None
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:
      None

      Description

      The current replication mechanism of SolrCloud is called state machine, which replicas start in same initial state and for each input, the input is distributed across replicas so all replicas will end up with same next state.

      But this type of replication have some drawbacks

      • The commit (which costly) have to run on all replicas
      • Slow recovery, because if replica miss more than N updates on its down time, the replica have to download entire index from its leader.

      So we create create another replication mode for SolrCloud called state transfer, which acts like master/slave replication. In basically

      • Leader distribute the update to other replicas, but the leader only apply the update to IW, other replicas just store the update to UpdateLog (act like replication).
      • Replicas frequently polling the latest segments from leader.

      Pros:

      • Lightweight for indexing, because only leader are running the commit, updates.
      • Very fast recovery, replicas just have to download the missing segments.

      On CAP point of view, this ticket will trying to promise to end users a distributed systems :

      • Partition tolerance
      • Weak Consistency for normal query : clusters can serve stale data. This happen when leader finish a commit and slave is fetching for latest segment. This period can at most pollInterval + time to fetch latest segment.
      • Consistency for RTG : if we do not use DQBs, replicas will consistence with master just like original SolrCloud mode
      • Weak Availability : just like original SolrCloud mode. If a leader down, client must wait until new leader being elected.

      To use this new replication mode, a new collection must be created with an additional parameter liveReplicas=1

      http://localhost:8983/solr/admin/collections?action=CREATE&name=newCollection&numShards=2&replicationFactor=1&realtimeReplicas=1
      

        Attachments

        1. OnlyLeaderIndexesTest-fail-1.zip
          79 kB
          Mark Miller
        2. SOLR-9835.patch
          96 kB
          Cao Manh Dat
        3. SOLR-9835.patch
          93 kB
          Cao Manh Dat
        4. SOLR-9835.patch
          91 kB
          Cao Manh Dat
        5. SOLR-9835.patch
          91 kB
          Cao Manh Dat
        6. SOLR-9835.patch
          88 kB
          Cao Manh Dat
        7. SOLR-9835.patch
          88 kB
          Cao Manh Dat
        8. SOLR-9835.patch
          77 kB
          Cao Manh Dat
        9. SOLR-9835.patch
          77 kB
          Cao Manh Dat
        10. SOLR-9835.patch
          80 kB
          Cao Manh Dat
        11. SOLR-9835.patch
          84 kB
          Cao Manh Dat
        12. SOLR-9835.patch
          86 kB
          Cao Manh Dat
        13. SOLR-9835.patch
          78 kB
          Cao Manh Dat
        14. SOLR-9835.patch
          74 kB
          Cao Manh Dat
        15. SOLR-9835.patch
          51 kB
          Cao Manh Dat
        16. SOLR-9835.patch
          51 kB
          Cao Manh Dat

          Issue Links

            Activity

              People

              • Assignee:
                shalinmangar Shalin Shekhar Mangar
                Reporter:
                caomanhdat Cao Manh Dat
              • Votes:
                2 Vote for this issue
                Watchers:
                17 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: