Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-10070

HBase read high-availability using timeline-consistent region replicas

    XMLWordPrintableJSON

Details

    Description

      In the present HBase architecture, it is hard, probably impossible, to satisfy constraints like 99th percentile of the reads will be served under 10 ms. One of the major factors that affects this is the MTTR for regions. There are three phases in the MTTR process - detection, assignment, and recovery. Of these, the detection is usually the longest and is presently in the order of 20-30 seconds. During this time, the clients would not be able to read the region data.

      However, some clients will be better served if regions will be available for reads during recovery for doing eventually consistent reads. This will help with satisfying low latency guarantees for some class of applications which can work with stale reads.

      For improving read availability, we propose a replicated read-only region serving design, also referred as secondary regions, or region shadows. Extending current model of a region being opened for reads and writes in a single region server, the region will be also opened for reading in region servers. The region server which hosts the region for reads and writes (as in current case) will be declared as PRIMARY, while 0 or more region servers might be hosting the region as SECONDARY. There may be more than one secondary (replica count > 2).

      Will attach a design doc shortly which contains most of the details and some thoughts about development approaches. Reviews are more than welcome.

      We also have a proof of concept patch, which includes the master and regions server side of changes. Client side changes will be coming soon as well.

      Attachments

        Issue Links

          incorporates

          New Feature - A new feature of the product, which has yet to be developed. HBASE-13063 Allow to turn off memstore replication for region replicas

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          is related to

          Sub-task - The sub-task of the issue HBASE-10703 TestAsyncProcess does not pass on HBASE-10070

          • Major - Major loss of function.
          • Closed

          Sub-task - The sub-task of the issue HBASE-10791 Add integration test to demonstrate performance improvement

          • Major - Major loss of function.
          • Closed

          Sub-task - The sub-task of the issue HBASE-10817 Add some tests on a real cluster for replica: multi master, replication

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HBASE-18070 Enable memstore replication for meta replica

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HBASE-13063 Allow to turn off memstore replication for region replicas

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-10605 Manage the call timeout in the server

          • Major - Major loss of function.
          • Closed
          relates to

          Sub-task - The sub-task of the issue HBASE-11367 Pluggable replication endpoint

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HBASE-2357 Coprocessors: Add read-only region replicas (slaves) for availability and fast region recovery

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-10785 Metas own location should be cached

          • Major - Major loss of function.
          • Closed
          requires

          Sub-task - The sub-task of the issue HBASE-10957 HBASE-10070: HMaster can abort with NPE in #rebuildUserRegions

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-10525 Allow the client to use a different thread for writing to ease interrupt

          • Major - Major loss of function.
          • Closed
          links to

          Web Link Region API design for HBASE-10070

          1.
          		 HRegionInfo changes for adding replicaId and MetaEditor/MetaReader changes for region replicas Sub-task Closed Enis Soztutar
          2.
          		 HTableDescriptor changes for region replicas Sub-task Closed Devaraj Das
          3.
          		 Master/AM/RegionStates changes to create and assign region replicas Sub-task Closed Devaraj Das
          4.
          		 LoadBalancer changes for supporting region replicas Sub-task Closed Enis Soztutar
          5.
          		 Region and RegionServer changes for opening region replicas, and refreshing store files Sub-task Closed Enis Soztutar
          6.
          		 Add an API for defining consistency per request Sub-task Closed Enis Soztutar
          7.
          		 Failover RPC's from client using region replicas Sub-task Closed Nicolas Liochon
          8.
          		 Failover RPC's for multi-get Sub-task Closed Sergey Shelukhin
          9.
          		 Failover RPC's for scans Sub-task Closed Devaraj Das
          10.
          		 Master/RS WebUI changes for region replicas Sub-task Closed Devaraj Das
          11.
          		 Enable/AlterTable support for region replicas Sub-task Closed Devaraj Das
          12.
          		 HBCK changes for supporting region replicas Sub-task Closed Devaraj Das
          13.
          		 Provide user documentation for region replicas Sub-task Closed Enis Soztutar
          14.
          		 NPE in MetaCache.clearCache() Sub-task Closed Enis Soztutar
          15.
          		 Create an IntegrationTest for region replicas Sub-task Closed Enis Soztutar
          16.
          		 Integration test for multi-get calls Sub-task Closed Devaraj Das
          17.
          		 LoadBalancer.needsBalance() should check for co-located region replicas as well Sub-task Closed Devaraj Das
          18.
          		 NullPointerException in ConnectionManager$HConnectionImplementation.locateRegionInMeta() due to missing region info Sub-task Closed Ted Yu
          19.
          		 StoreFileRefresherChore throws ConcurrentModificationException sometimes Sub-task Closed Devaraj Das
          20.
          		 Multiget doesn't fully work Sub-task Closed Sergey Shelukhin
          21.
          		 TestStochasticLoadBalancer.testRegionReplicationOnMidClusterWithRacks() is flaky Sub-task Closed Enis Soztutar
          22.
          		 Table snapshot should handle tables whose REGION_REPLICATION is greater than one Sub-task Closed Devaraj Das
          23.
          		 HBCK should be updated to do replica related checks Sub-task Closed Devaraj Das
          24.
          		 Cache invalidation improvements from client side Sub-task Closed Enis Soztutar
          25.
          		 BaseLoadBalancer#roundRobinAssignment() may add same region to assignment plan multiple times Sub-task Closed Ted Yu
          26.
          		 TestAsyncProcess does not pass on HBASE-10070 Sub-task Closed Nicolas Liochon
          27.
          		 Enable table doesn't balance out replicas evenly if the replicas were unassigned earlier Sub-task Closed Devaraj Das
          28.
          		 Fix RegionStates.getRegionAssignments to not add duplicate regions Sub-task Closed Devaraj Das
          29.
          		 Replica map update is problematic in RegionStates Sub-task Closed Devaraj Das
          30.
          		 Unique keys accounting in MultiThreadedReader is incorrect Sub-task Closed Ted Yu
          31.
          		 Add integration test to demonstrate performance improvement Sub-task Closed Nick Dimiduk
          32.
          		 multi-get should handle replica location missing from cache Sub-task Closed Sergey Shelukhin
          33.
          		 LoadTestTool should share the connection and connection pool Sub-task Closed Enis Soztutar
          34.
          		 Add integration test for bulkload with replicas Sub-task Closed Devaraj Das
          35.
          		 Add some tests on a real cluster for replica: multi master, replication Sub-task Closed Nicolas Liochon
          36.
          		 TestRegionRebalancing is failing Sub-task Closed Enis Soztutar
          37.
          		 Use HFileLink in opening region files from secondaries Sub-task Closed Enis Soztutar
          38.
          		 support parallel request cancellation for multi-get Sub-task Closed Devaraj Das
          39.
          		 HBASE-10070: HMaster can abort with NPE in #rebuildUserRegions Sub-task Closed Nicolas Liochon
          40.
          		 TestFromClientSideWithCoprocessor#testGetClosestRowBefore fails due to invalid block size Sub-task Closed Unassigned
          41.
          		 Fixes for scans on a replicated table Sub-task Closed Devaraj Das
          42.
          		 Timeline Consistent region replicas - Phase 2 design Sub-task Closed Enis Soztutar
          43.
          		 Handle splitting/merging of regions that have region_replication greater than one Sub-task Closed Devaraj Das
          44.
          		 Fix for metas location cache from HBASE-10785 Sub-task Closed Enis Soztutar
          45.
          		 Write flush events to WAL Sub-task Closed Enis Soztutar
          46.
          		 Write region open/close events to WAL Sub-task Closed Enis Soztutar
          47.
          		 Write bulk load COMMIT events to WAL Sub-task Closed Alex Newman
          48.
          		 Async WAL replication for region replicas Sub-task Closed Enis Soztutar
          49.
          		 Flush / Compaction handling from secondary region replicas Sub-task Closed Enis Soztutar
          50.
          		 Bulk load handling from secondary region replicas Sub-task Closed Jeffrey Zhong
          51.
          		 RegionLocations::getRegionLocation can return unexpected replica Sub-task Closed Unassigned
          52.
          		 Add support for doing get/scans against a particular replica_id Sub-task Closed Jeffrey Zhong
          53.
          		 hbase:meta's regions can be replicated Sub-task Closed Devaraj Das
          54.
          		 Failover handling for secondary region replicas Sub-task Closed Enis Soztutar
          55.
          		 Integration test for async wal replication to secondary regions Sub-task Closed Enis Soztutar
          56.
          		 Directly invoking split & merge of replica regions should be disallowed Sub-task Closed Devaraj Das
          57.
          		 Region replicas should be added to the meta table at the time of table creation Sub-task Closed Enis Soztutar
          58.
          		 Improve cancellation for the scan RPCs Sub-task Closed Devaraj Das
          59.
          		 Maintain SeqId monotonically increasing Sub-task Closed Jeffrey Zhong
          60.
          		 Replicas of regions can be cached from different instances of the table in MetaCache Sub-task Closed Enis Soztutar
          61.
          		 Handling memory pressure for secondary region replicas Sub-task Closed Enis Soztutar
          62.
          		 RegionReplicaReplicationEndpoint should not set the RPC Codec Sub-task Closed Enis Soztutar
          63.
          		 Disallow non-atomic update operations when TIMELINE consistency is enabled Sub-task Closed Unassigned
          64.
          		 Async wal replication for region replicas and dist log replay does not work together Sub-task Closed Enis Soztutar
          65.
          		 ModifyTable increasing the region replica count should also auto-setup RRRE Sub-task Closed Enis Soztutar
          66.
          		 Split out locality metrics among primary and secondary region Sub-task Closed Ted Yu
          67.
          		 Handle FileNotFoundException in region replica replay for flush/compaction events Sub-task Closed Enis Soztutar
          68.
          		 Update documentation for 10070 Phase 2 changes Sub-task Closed Enis Soztutar

          Activity

            People

              enis Enis Soztutar
              enis Enis Soztutar
              Votes:
              11 Vote for this issue
              Watchers:
              102 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: