Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-15867

Move HBase replication tracking from ZooKeeper to HBase

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.1.0
    • Fix Version/s: 2.3.0
    • Component/s: Replication
    • Labels:
      None
    • Flags:
      Patch

      Description

      Move the WAL file and offset tracking out of ZooKeeper and into an HBase table called hbase:replication.
      The largest three new changes will be two classes ReplicationTableBase, TableBasedReplicationQueues, and TableBasedReplicationQueuesClient. As of now ReplicationPeers and HFileRef's tracking will not be implemented. Subtasks have been filed for these two jobs.

        Attachments

          Issue Links

          1.
          Refactor ReplicationQueues.isThisOurZnode() to ReplicationQueues.isThisOurRegionServer Sub-task Resolved Joseph
          2.
          Adding WAL files and tracking offsets in HBase. Sub-task Resolved Joseph
          3.
          Add Javadocs for ReplicationQueuesHBaseImpl Sub-task Resolved Joseph
          4.
          Modify queue naming standards to speed up ReplicationQueuesHBaseImpl Sub-task Resolved Joseph
          5.
          Implement ClaimQueues on top of HBase Sub-task Resolved Joseph
          6.
          Create a ReplicationQueuesClientHBaseImpl Sub-task Resolved Joseph
          7.
          Fix ReplicationTableBase initialization to make it non-blocking Sub-task Resolved Joseph
          8.
          Fix flakey tests Sub-task Resolved Joseph
          9.
          Fix table based replication related configs Sub-task Resolved Joseph
          10.
          Make TableBasedReplicationQueuesImpl check that queue exists before attempting to remove it Sub-task Resolved Joseph
          11.
          Add in UI description of Replication Table Sub-task Resolved Joseph
          12.
          Figure out retry limit and timing for replication queue table operations Sub-task Patch Available Ashu Pachauri
          13.
          Cannot open regions after non-graceful shutdown due to deadlock with Replication Table Sub-task Patch Available Ashu Pachauri
          14.
          Backport replication tracking via system table to Branch-1 Sub-task Resolved Ashu Pachauri
          15.
          Implement HFile ref's tracking (bulk loading) in ReplicationQueuesHBaseImpl and ReplicationQueuesClientHBaseImpl Sub-task Resolved Zheng Hu
          16.
          Removing peers seem to be leaving spare queues Sub-task Resolved Unassigned
          17.
          Move Replication Peers into an HBase table too Sub-task Resolved Zheng Hu
          18.
          Metrics for overall replication backlog length Sub-task Resolved Zheng Hu
          19.
          Race Condition inside of TestReplicationStateHBase Unit Tests. Sub-task Open Zheng Hu
          20.
          Removing peers seems to be leaving spare queues Sub-task Resolved Guanghao Zhang
          21.
          Rename the KeeperException to ReplicationException in ReplicationQueuesClient for abstracting Sub-task Resolved Zheng Hu
          22.
          Use nonce and procedure v2 for add/remove replication peer Sub-task Resolved Unassigned
          23.
          Add table based replication peers/queues storage back Sub-task Resolved Zheng Hu
          24.
          Make sure the RS/Master can works fine when using table based replication storage layer Sub-task Open Zheng Hu
          25.
          Make the API with multi-operations in TableReplicationQueueStorage to be atomic Sub-task Open Zheng Hu
          26.
          Consider rolling update from zookeeper based replication storage to table based replication storage Sub-task Open Zheng Hu
          27.
          Don't split the region in hbase:replication to ensure that multi-mutations in a batch from replication storage to be atomic Sub-task Open Zheng Hu
          28.
          Introduce a global configuration to decide whether we use table or zk based replication storage Sub-task Open Zheng Hu
          29.
          Let all replication related UTs can work fine on both ZK based storage and Table based storage Sub-task Open Zheng Hu

            Activity

              People

              • Assignee:
                openinx Zheng Hu
                Reporter:
                Vegetable26 Joseph
              • Votes:
                0 Vote for this issue
                Watchers:
                26 Start watching this issue

                Dates

                • Due:
                  Created:
                  Updated: