Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-10070

HBase read high-availability using timeline-consistent region replicas

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      In the present HBase architecture, it is hard, probably impossible, to satisfy constraints like 99th percentile of the reads will be served under 10 ms. One of the major factors that affects this is the MTTR for regions. There are three phases in the MTTR process - detection, assignment, and recovery. Of these, the detection is usually the longest and is presently in the order of 20-30 seconds. During this time, the clients would not be able to read the region data.

      However, some clients will be better served if regions will be available for reads during recovery for doing eventually consistent reads. This will help with satisfying low latency guarantees for some class of applications which can work with stale reads.

      For improving read availability, we propose a replicated read-only region serving design, also referred as secondary regions, or region shadows. Extending current model of a region being opened for reads and writes in a single region server, the region will be also opened for reading in region servers. The region server which hosts the region for reads and writes (as in current case) will be declared as PRIMARY, while 0 or more region servers might be hosting the region as SECONDARY. There may be more than one secondary (replica count > 2).

      Will attach a design doc shortly which contains most of the details and some thoughts about development approaches. Reviews are more than welcome.

      We also have a proof of concept patch, which includes the master and regions server side of changes. Client side changes will be coming soon as well.

      Attachments

        Issue Links

        1.
        HRegionInfo changes for adding replicaId and MetaEditor/MetaReader changes for region replicas Sub-task Closed Enis Soztutar Actions
        2.
        HTableDescriptor changes for region replicas Sub-task Closed Devaraj Das Actions
        3.
        Master/AM/RegionStates changes to create and assign region replicas Sub-task Closed Devaraj Das Actions
        4.
        LoadBalancer changes for supporting region replicas Sub-task Closed Enis Soztutar Actions
        5.
        Region and RegionServer changes for opening region replicas, and refreshing store files Sub-task Closed Enis Soztutar Actions
        6.
        Add an API for defining consistency per request Sub-task Closed Enis Soztutar Actions
        7.
        Failover RPC's from client using region replicas Sub-task Closed Nicolas Liochon Actions
        8.
        Failover RPC's for multi-get Sub-task Closed Sergey Shelukhin Actions
        9.
        Failover RPC's for scans Sub-task Closed Devaraj Das Actions
        10.
        Master/RS WebUI changes for region replicas Sub-task Closed Devaraj Das Actions
        11.
        Enable/AlterTable support for region replicas Sub-task Closed Devaraj Das Actions
        12.
        HBCK changes for supporting region replicas Sub-task Closed Devaraj Das Actions
        13.
        Provide user documentation for region replicas Sub-task Closed Enis Soztutar Actions
        14.
        NPE in MetaCache.clearCache() Sub-task Closed Enis Soztutar Actions
        15.
        Create an IntegrationTest for region replicas Sub-task Closed Enis Soztutar Actions
        16.
        Integration test for multi-get calls Sub-task Closed Devaraj Das Actions
        17.
        LoadBalancer.needsBalance() should check for co-located region replicas as well Sub-task Closed Devaraj Das Actions
        18.
        NullPointerException in ConnectionManager$HConnectionImplementation.locateRegionInMeta() due to missing region info Sub-task Closed Ted Yu Actions
        19.
        StoreFileRefresherChore throws ConcurrentModificationException sometimes Sub-task Closed Devaraj Das Actions
        20.
        Multiget doesn't fully work Sub-task Closed Sergey Shelukhin Actions
        21.
        TestStochasticLoadBalancer.testRegionReplicationOnMidClusterWithRacks() is flaky Sub-task Closed Enis Soztutar Actions
        22.
        Table snapshot should handle tables whose REGION_REPLICATION is greater than one Sub-task Closed Devaraj Das Actions
        23.
        HBCK should be updated to do replica related checks Sub-task Closed Devaraj Das Actions
        24.
        Cache invalidation improvements from client side Sub-task Closed Enis Soztutar Actions
        25.
        BaseLoadBalancer#roundRobinAssignment() may add same region to assignment plan multiple times Sub-task Closed Ted Yu Actions
        26.
        TestAsyncProcess does not pass on HBASE-10070 Sub-task Closed Nicolas Liochon Actions
        27.
        Enable table doesn't balance out replicas evenly if the replicas were unassigned earlier Sub-task Closed Devaraj Das Actions
        28.
        Fix RegionStates.getRegionAssignments to not add duplicate regions Sub-task Closed Devaraj Das Actions
        29.
        Replica map update is problematic in RegionStates Sub-task Closed Devaraj Das Actions
        30.
        Unique keys accounting in MultiThreadedReader is incorrect Sub-task Closed Ted Yu Actions
        31.
        Add integration test to demonstrate performance improvement Sub-task Closed Nick Dimiduk Actions
        32.
        multi-get should handle replica location missing from cache Sub-task Closed Sergey Shelukhin Actions
        33.
        LoadTestTool should share the connection and connection pool Sub-task Closed Enis Soztutar Actions
        34.
        Add integration test for bulkload with replicas Sub-task Closed Devaraj Das Actions
        35.
        Add some tests on a real cluster for replica: multi master, replication Sub-task Closed Nicolas Liochon Actions
        36.
        TestRegionRebalancing is failing Sub-task Closed Enis Soztutar Actions
        37.
        Use HFileLink in opening region files from secondaries Sub-task Closed Enis Soztutar Actions
        38.
        support parallel request cancellation for multi-get Sub-task Closed Devaraj Das Actions
        39.
        HBASE-10070: HMaster can abort with NPE in #rebuildUserRegions Sub-task Closed Nicolas Liochon Actions
        40.
        TestFromClientSideWithCoprocessor#testGetClosestRowBefore fails due to invalid block size Sub-task Closed Unassigned Actions
        41.
        Fixes for scans on a replicated table Sub-task Closed Devaraj Das Actions
        42.
        Timeline Consistent region replicas - Phase 2 design Sub-task Closed Enis Soztutar Actions
        43.
        Handle splitting/merging of regions that have region_replication greater than one Sub-task Closed Devaraj Das Actions
        44.
        Fix for metas location cache from HBASE-10785 Sub-task Closed Enis Soztutar Actions
        45.
        Write flush events to WAL Sub-task Closed Enis Soztutar Actions
        46.
        Write region open/close events to WAL Sub-task Closed Enis Soztutar Actions
        47.
        Write bulk load COMMIT events to WAL Sub-task Closed Alex Newman Actions
        48.
        Async WAL replication for region replicas Sub-task Closed Enis Soztutar Actions
        49.
        Flush / Compaction handling from secondary region replicas Sub-task Closed Enis Soztutar Actions
        50.
        Bulk load handling from secondary region replicas Sub-task Closed Jeffrey Zhong Actions
        51.
        RegionLocations::getRegionLocation can return unexpected replica Sub-task Closed Unassigned Actions
        52.
        Add support for doing get/scans against a particular replica_id Sub-task Closed Jeffrey Zhong Actions
        53.
        hbase:meta's regions can be replicated Sub-task Closed Devaraj Das Actions
        54.
        Failover handling for secondary region replicas Sub-task Closed Enis Soztutar Actions
        55.
        Integration test for async wal replication to secondary regions Sub-task Closed Enis Soztutar Actions
        56.
        Directly invoking split & merge of replica regions should be disallowed Sub-task Closed Devaraj Das Actions
        57.
        Region replicas should be added to the meta table at the time of table creation Sub-task Closed Enis Soztutar Actions
        58.
        Improve cancellation for the scan RPCs Sub-task Closed Devaraj Das Actions
        59.
        Maintain SeqId monotonically increasing Sub-task Closed Jeffrey Zhong Actions
        60.
        Replicas of regions can be cached from different instances of the table in MetaCache Sub-task Closed Enis Soztutar Actions
        61.
        Handling memory pressure for secondary region replicas Sub-task Closed Enis Soztutar Actions
        62.
        RegionReplicaReplicationEndpoint should not set the RPC Codec Sub-task Closed Enis Soztutar Actions
        63.
        Disallow non-atomic update operations when TIMELINE consistency is enabled Sub-task Closed Unassigned Actions
        64.
        Async wal replication for region replicas and dist log replay does not work together Sub-task Closed Enis Soztutar Actions
        65.
        ModifyTable increasing the region replica count should also auto-setup RRRE Sub-task Closed Enis Soztutar Actions
        66.
        Split out locality metrics among primary and secondary region Sub-task Closed Ted Yu Actions
        67.
        Handle FileNotFoundException in region replica replay for flush/compaction events Sub-task Closed Enis Soztutar Actions
        68.
        Update documentation for 10070 Phase 2 changes Sub-task Closed Enis Soztutar Actions

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            enis Enis Soztutar
            enis Enis Soztutar
            Votes:
            11 Vote for this issue
            Watchers:
            108 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment