HBase
  1. HBase
  2. HBASE-10070

HBase read high-availability using timeline-consistent region replicas

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 2.0.0, 1.2.0
    • Component/s: None
    • Labels:
      None

      Description

      In the present HBase architecture, it is hard, probably impossible, to satisfy constraints like 99th percentile of the reads will be served under 10 ms. One of the major factors that affects this is the MTTR for regions. There are three phases in the MTTR process - detection, assignment, and recovery. Of these, the detection is usually the longest and is presently in the order of 20-30 seconds. During this time, the clients would not be able to read the region data.

      However, some clients will be better served if regions will be available for reads during recovery for doing eventually consistent reads. This will help with satisfying low latency guarantees for some class of applications which can work with stale reads.

      For improving read availability, we propose a replicated read-only region serving design, also referred as secondary regions, or region shadows. Extending current model of a region being opened for reads and writes in a single region server, the region will be also opened for reading in region servers. The region server which hosts the region for reads and writes (as in current case) will be declared as PRIMARY, while 0 or more region servers might be hosting the region as SECONDARY. There may be more than one secondary (replica count > 2).

      Will attach a design doc shortly which contains most of the details and some thoughts about development approaches. Reviews are more than welcome.

      We also have a proof of concept patch, which includes the master and regions server side of changes. Client side changes will be coming soon as well.

        Issue Links

        1.
        HRegionInfo changes for adding replicaId and MetaEditor/MetaReader changes for region replicas Sub-task Closed Enis Soztutar
         
        2.
        HTableDescriptor changes for region replicas Sub-task Closed Devaraj Das
         
        3.
        Master/AM/RegionStates changes to create and assign region replicas Sub-task Closed Devaraj Das
         
        4.
        LoadBalancer changes for supporting region replicas Sub-task Closed Enis Soztutar
         
        5.
        Region and RegionServer changes for opening region replicas, and refreshing store files Sub-task Closed Enis Soztutar
         
        6.
        Add an API for defining consistency per request Sub-task Closed Enis Soztutar
         
        7.
        Failover RPC's from client using region replicas Sub-task Closed Nicolas Liochon
         
        8.
        Failover RPC's for multi-get Sub-task Closed Sergey Shelukhin
         
        9.
        Failover RPC's for scans Sub-task Closed Devaraj Das
         
        10. Shell changes for setting consistency per request Sub-task Open Enis Soztutar
         
        11.
        Master/RS WebUI changes for region replicas Sub-task Closed Devaraj Das
         
        12.
        Enable/AlterTable support for region replicas Sub-task Closed Devaraj Das
         
        13.
        HBCK changes for supporting region replicas Sub-task Closed Devaraj Das
         
        14.
        Provide user documentation for region replicas Sub-task Closed Enis Soztutar
         
        15.
        NPE in MetaCache.clearCache() Sub-task Closed Enis Soztutar
         
        16.
        Create an IntegrationTest for region replicas Sub-task Closed Enis Soztutar
         
        17.
        Integration test for multi-get calls Sub-task Closed Devaraj Das
         
        18.
        LoadBalancer.needsBalance() should check for co-located region replicas as well Sub-task Closed Devaraj Das
         
        19.
        NullPointerException in ConnectionManager$HConnectionImplementation.locateRegionInMeta() due to missing region info Sub-task Closed Ted Yu
         
        20.
        StoreFileRefresherChore throws ConcurrentModificationException sometimes Sub-task Closed Devaraj Das
         
        21.
        Multiget doesn't fully work Sub-task Closed Sergey Shelukhin
         
        22.
        TestStochasticLoadBalancer.testRegionReplicationOnMidClusterWithRacks() is flaky Sub-task Closed Enis Soztutar
         
        23.
        Table snapshot should handle tables whose REGION_REPLICATION is greater than one Sub-task Closed Devaraj Das
         
        24.
        HBCK should be updated to do replica related checks Sub-task Resolved Devaraj Das
         
        25.
        Cache invalidation improvements from client side Sub-task Closed Enis Soztutar
         
        26.
        BaseLoadBalancer#roundRobinAssignment() may add same region to assignment plan multiple times Sub-task Closed Ted Yu
         
        27.
        TestAsyncProcess does not pass on HBASE-10070 Sub-task Resolved Nicolas Liochon
         
        28.
        Enable table doesn't balance out replicas evenly if the replicas were unassigned earlier Sub-task Closed Devaraj Das
         
        29.
        Fix RegionStates.getRegionAssignments to not add duplicate regions Sub-task Closed Devaraj Das
         
        30.
        Replica map update is problematic in RegionStates Sub-task Closed Devaraj Das
         
        31.
        Unique keys accounting in MultiThreadedReader is incorrect Sub-task Closed Ted Yu
         
        32.
        Add integration test to demonstrate performance improvement Sub-task Closed Nick Dimiduk
         
        33.
        multi-get should handle replica location missing from cache Sub-task Closed Sergey Shelukhin
         
        34.
        LoadTestTool should share the connection and connection pool Sub-task Closed Enis Soztutar
         
        35.
        Add integration test for bulkload with replicas Sub-task Closed Devaraj Das
         
        36.
        Add some tests on a real cluster for replica: multi master, replication Sub-task Closed Nicolas Liochon
         
        37.
        TestRegionRebalancing is failing Sub-task Closed Enis Soztutar
         
        38.
        Use HFileLink in opening region files from secondaries Sub-task Closed Enis Soztutar
         
        39.
        support parallel request cancellation for multi-get Sub-task Resolved Devaraj Das
         
        40.
        HBASE-10070: HMaster can abort with NPE in #rebuildUserRegions Sub-task Closed Nicolas Liochon
         
        41.
        TestFromClientSideWithCoprocessor#testGetClosestRowBefore fails due to invalid block size Sub-task Resolved Ted Yu
         
        42.
        Fixes for scans on a replicated table Sub-task Closed Devaraj Das
         
        43. Timeline Consistent region replicas - Phase 2 design Sub-task Open Enis Soztutar
         
        44.
        Handle splitting/merging of regions that have region_replication greater than one Sub-task Resolved Devaraj Das
         
        45.
        Fix for metas location cache from HBASE-10785 Sub-task Closed Enis Soztutar
         
        46.
        Write flush events to WAL Sub-task Closed Enis Soztutar
         
        47.
        Write region open/close events to WAL Sub-task Closed Enis Soztutar
         
        48.
        Write bulk load COMMIT events to WAL Sub-task Resolved Alex Newman
         
        49.
        Async WAL replication for region replicas Sub-task Resolved Enis Soztutar
         
        50.
        Flush / Compaction handling from secondary region replicas Sub-task Resolved Enis Soztutar
         
        51.
        Bulk load handling from secondary region replicas Sub-task Resolved Jeffrey Zhong
         
        52.
        RegionLocations::getRegionLocation can return unexpected replica Sub-task Resolved Unassigned
         
        53.
        Add support for doing get/scans against a particular replica_id Sub-task Closed Jeffrey Zhong
         
        54.
        hbase:meta's regions can be replicated Sub-task Resolved Devaraj Das
         
        55.
        Failover handling for secondary region replicas Sub-task Resolved Enis Soztutar
         
        56.
        Integration test for async wal replication to secondary regions Sub-task Resolved Enis Soztutar
         
        57.
        Directly invoking split & merge of replica regions should be disallowed Sub-task Resolved Devaraj Das
         
        58.
        Region replicas should be added to the meta table at the time of table creation Sub-task Resolved Enis Soztutar
         
        59.
        Improve cancellation for the scan RPCs Sub-task Resolved Devaraj Das
         
        60.
        Maintain SeqId monotonically increasing Sub-task Closed Jeffrey Zhong
         
        61.
        Replicas of regions can be cached from different instances of the table in MetaCache Sub-task Resolved Enis Soztutar
         
        62.
        Handling memory pressure for secondary region replicas Sub-task Resolved Enis Soztutar
         
        63.
        RegionReplicaReplicationEndpoint should not set the RPC Codec Sub-task Resolved Enis Soztutar
         
        64.
        Disallow non-atomic update operations when TIMELINE consistency is enabled Sub-task Resolved Unassigned
         
        65.
        Async wal replication for region replicas and dist log replay does not work together Sub-task Resolved Enis Soztutar
         
        66.
        ModifyTable increasing the region replica count should also auto-setup RRRE Sub-task Resolved Enis Soztutar
         
        67.
        Split out locality metrics among primary and secondary region Sub-task Resolved Ted Yu
         
        68.
        Handle FileNotFoundException in region replica replay for flush/compaction events Sub-task Resolved Enis Soztutar
         

          Activity

            People

            • Assignee:
              Enis Soztutar
              Reporter:
              Enis Soztutar
            • Votes:
              11 Vote for this issue
              Watchers:
              87 Start watching this issue

              Dates

              • Created:
                Updated:

                Development