Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 3.4.0
    • Fix Version/s: 3.6.0
    • Component/s: server
    • Labels:
      None
    • Environment:
    • Release Note:
      ZOOKEEPER-892. Remote replication of ZooKeeper data (Anirban Roy)
    • Tags:
      zkrepl replication zoorepl

      Description

      ZooKeeper is a highly available and scalable system for distributed synchrony and is frequently used for cluster management. In its current incarnation it has issues with communication and data replication across extended geographic locations. Presently, the only way to distribute ZooKeeper across multiple data centers is to maintain a cross-colo Quorum using Observer members, leading to unnecessary consumption of bandwidth and performance impacts. As the title suggests, this work aims to to provide replication of ZooKeeper data from one site to others using a new type of ZooKeeper member called a Publisher. The broad idea is to have a complete instance of the current ZooKeeper at each geographic location in a master-slave setup. The Publisher will be a part of the Master ZooKeeper Site and will push changes to a FIFO queue and make it available to any interested client. The slave ZooKeeper runs client application called Replicator which receives and replays the changes to slave instance. Multiple slave Replicators can subscribes to the master Publisher and receive changes with guaranteed ordering. It will be asynchronous, non-intrusive, loosely-coupled and can be applied to a subset of the data. This scheme will bring about many of the benefits of database replication such as resilience to site failure and localized serving across data centers. In short, the goal is to provide remote (sub-tree) data replication with guaranteed ordering, without affecting the Master ZooKeeper performance.

      1. zoorepl.patch
        176 kB
        Anirban Roy
      2. zoorepl.patch
        176 kB
        Anirban Roy
      3. zoorepl.pdf
        548 kB
        Anirban Roy

        Activity

        Michi Mutsuzaki made changes -
        Fix Version/s 3.6.0 [ 12326518 ]
        Fix Version/s 3.5.0 [ 12316644 ]
        Flavio Junqueira made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Mahadev konar made changes -
        Fix Version/s 3.5.0 [ 12316644 ]
        Fix Version/s 3.4.0 [ 12314469 ]
        Anirban Roy made changes -
        Status In Progress [ 3 ] Patch Available [ 10002 ]
        Anirban Roy made changes -
        Status Open [ 1 ] In Progress [ 3 ]
        Anirban Roy made changes -
        Attachment zoorepl.patch [ 12480083 ]
        Patrick Hunt made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Mahadev konar made changes -
        Status Reopened [ 4 ] Patch Available [ 10002 ]
        Mahadev konar made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Anirban Roy made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Anirban Roy made changes -
        Attachment zoorepl.patch [ 12478364 ]
        Anirban Roy made changes -
        Attachment zoorepl.patch [ 12479547 ]
        Anirban Roy made changes -
        Component/s server [ 12312382 ]
        Component/s quorum [ 12312379 ]
        Anirban Roy made changes -
        Attachment zoorepl.patch [ 12478034 ]
        Anirban Roy made changes -
        Attachment zoorepl.patch [ 12478125 ]
        Anirban Roy made changes -
        Attachment zoorepl.patch [ 12478243 ]
        Anirban Roy made changes -
        Attachment zoorepl.patch [ 12478364 ]
        Anirban Roy made changes -
        Status In Progress [ 3 ] Patch Available [ 10002 ]
        Anirban Roy made changes -
        Status Open [ 1 ] In Progress [ 3 ]
        Anirban Roy made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Anirban Roy made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Anirban Roy made changes -
        Status In Progress [ 3 ] Open [ 1 ]
        Anirban Roy made changes -
        Attachment zoorepl.patch [ 12478243 ]
        Anirban Roy made changes -
        Due Date 2011-04-15 00:00:00.0 2011-05-15 00:00:00.0
        Anirban Roy made changes -
        Status Patch Available [ 10002 ] In Progress [ 3 ]
        Anirban Roy made changes -
        Attachment zoorepl.patch [ 12477395 ]
        Anirban Roy made changes -
        Attachment zoorepl.patch [ 12475782 ]
        Anirban Roy made changes -
        Attachment zoorepl.patch [ 12475688 ]
        Anirban Roy made changes -
        Attachment zoorepl.patch [ 12475339 ]
        Anirban Roy made changes -
        Attachment zoorepl.patch [ 12478125 ]
        Anirban Roy made changes -
        Attachment zoorepl.patch [ 12478034 ]
        Anirban Roy made changes -
        Attachment zoorepl.patch [ 12477395 ]
        Anirban Roy made changes -
        Attachment zoorepl.patch [ 12475782 ]
        Anirban Roy made changes -
        Attachment zoorepl.patch [ 12475688 ]
        Anirban Roy made changes -
        Attachment zoorepl.pdf [ 12475340 ]
        Anirban Roy made changes -
        Attachment zoorepl.patch [ 12475339 ]
        Anirban Roy made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Release Note ZOOKEEPER-892. Remote replication of ZooKeeper data (Anirban Roy)
        Anirban Roy made changes -
        Field Original Value New Value
        Tags zkrepl replication zkrepl replication zoorepl
        Description ZooKeeper is a highly available and scalable system for distributed synchrony and is used for cluster management. In its current incarnation it has issues with cross-colo communication and data replication. Presently, the only way to distribute ZooKeeper across multiple data centers is to maintain a cross-colo Quorum using Observer members, leading to huge bandwidth consumption and performance degradation. The idea behind the ZooKeeper replication feature is to provide replication of ZooKeeper data from one site to others using a new type of ZooKeeper member called a Replicator. The Replicator will be asynchronous, non-intrusive, and can be applied to a subset of the data. It will be a part of the Main ZooKeeper Site and will push changes to multiple data centers with guaranteed ordering of events. This will bring about many of the benefits of database replication such as resilience to site failure and localized serving across data centers. In short, the goal is to provide remote (sub) data replication with guaranteed ordering, without affecting the Main ZooKeeper performance. ZooKeeper is a highly available and scalable system for distributed synchrony and is frequently used for cluster management. In its current incarnation it has issues with communication and data replication across extended geographic locations. Presently, the only way to distribute ZooKeeper across multiple data centers is to maintain a cross-colo Quorum using Observer members, leading to unnecessary consumption of bandwidth and performance impacts. As the title suggests, this work aims to to provide replication of ZooKeeper data from one site to others using a new type of ZooKeeper member called a Publisher. The broad idea is to have a complete instance of the current ZooKeeper at each geographic location in a master-slave setup. The Publisher will be a part of the Master ZooKeeper Site and will push changes to a FIFO queue and make it available to any interested client. The slave ZooKeeper runs client application called Replicator which receives and replays the changes to slave instance. Multiple slave Replicators can subscribes to the master Publisher and receive changes with guaranteed ordering. It will be asynchronous, non-intrusive, loosely-coupled and can be applied to a subset of the data. This scheme will bring about many of the benefits of database replication such as resilience to site failure and localized serving across data centers. In short, the goal is to provide remote (sub-tree) data replication with guaranteed ordering, without affecting the Master ZooKeeper performance.
        Due Date 2011-01-31 05:00:00.0 2011-04-15 00:00:00.0
        Anirban Roy created issue -

          People

          • Assignee:
            Anirban Roy
            Reporter:
            Anirban Roy
          • Votes:
            0 Vote for this issue
            Watchers:
            16 Start watching this issue

            Dates

            • Due:
              Created:
              Updated:

              Time Tracking

              Estimated:
              Original Estimate - 2,688h
              2,688h
              Remaining:
              Remaining Estimate - 2,688h
              2,688h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development