[HBASE-10070] HBase read high-availability using timeline-consistent region replicas

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0, 2.0.0
• Component/s: Admin, API, LatencyResilience
• Labels: needs_releasenote
• Assignee: Enis Soztutar
• Reporter: Enis Soztutar
• Created: 03/Dec/13 00:44
• Updated: 17/Jun/22 04:48
• Resolved: 26/Jun/15 22:33

Description

In the present HBase architecture, it is hard, probably impossible, to satisfy constraints like 99th percentile of the reads will be served under 10 ms. One of the major factors that affects this is the MTTR for regions. There are three phases in the MTTR process - detection, assignment, and recovery. Of these, the detection is usually the longest and is presently in the order of 20-30 seconds. During this time, the clients would not be able to read the region data.

However, some clients will be better served if regions will be available for reads during recovery for doing eventually consistent reads. This will help with satisfying low latency guarantees for some class of applications which can work with stale reads.

For improving read availability, we propose a replicated read-only region serving design, also referred as secondary regions, or region shadows. Extending current model of a region being opened for reads and writes in a single region server, the region will be also opened for reading in region servers. The region server which hosts the region for reads and writes (as in current case) will be declared as PRIMARY, while 0 or more region servers might be hosting the region as SECONDARY. There may be more than one secondary (replica count > 2).

Will attach a design doc shortly which contains most of the details and some thoughts about development approaches. Reviews are more than welcome.

We also have a proof of concept patch, which includes the master and regions server side of changes. Client side changes will be coming soon as well.

Issue Links

• incorporates: HBASE-13063 Allow to turn off memstore replication for region replicas
• is related to: HBASE-10703 TestAsyncProcess does not pass on HBASE-10070
• is related to: HBASE-10791 Add integration test to demonstrate performance improvement
• is related to: HBASE-10817 Add some tests on a real cluster for replica: multi master, replication
• is related to: HBASE-18070 Enable memstore replication for meta replica
• is related to: HBASE-13063 Allow to turn off memstore replication for region replicas
• is related to: HBASE-10605 Manage the call timeout in the server
• relates to: HBASE-11367 Pluggable replication endpoint
• relates to: HBASE-2357 Coprocessors: Add read-only region replicas (slaves) for availability and fast region recovery
• relates to: HBASE-10785 Metas own location should be cached
• requires: HBASE-10957 HBASE-10070: HMaster can abort with NPE in #rebuildUserRegions
• requires: HBASE-10525 Allow the client to use a different thread for writing to ease interrupt
• links to: Region API design for HBASE-10070

Sub-Tasks

• HRegionInfo changes for adding replicaId and MetaEditor/MetaReader changes for region replicas: Closed
• HTableDescriptor changes for region replicas: Closed
• Master/AM/RegionStates changes to create and assign region replicas: Closed
• LoadBalancer changes for supporting region replicas: Closed
• Region and RegionServer changes for opening region replicas, and refreshing store files: Closed
• Add an API for defining consistency per request: Closed
• Failover RPC's from client using region replicas: Closed
• Failover RPC's for multi-get: Closed
• Failover RPC's for scans: Closed
• Master/RS WebUI changes for region replicas: Closed
• Enable/AlterTable support for region replicas: Closed
• HBCK changes for supporting region replicas: Closed
• Provide user documentation for region replicas: Closed
• NPE in MetaCache.clearCache(): Closed
• Create an IntegrationTest for region replicas: Closed
• Integration test for multi-get calls: Closed
• LoadBalancer.needsBalance() should check for co-located region replicas as well: Closed
• NullPointerException in ConnectionManager$HConnectionImplementation.locateRegionInMeta() due to missing region info: Closed
• StoreFileRefresherChore throws ConcurrentModificationException sometimes: Closed
• Multiget doesn't fully work: Closed
• TestStochasticLoadBalancer.testRegionReplicationOnMidClusterWithRacks() is flaky: Closed
• Table snapshot should handle tables whose REGION_REPLICATION is greater than one: Closed
• HBCK should be updated to do replica related checks: Closed
• Cache invalidation improvements from client side: Closed
• BaseLoadBalancer#roundRobinAssignment() may add same region to assignment plan multiple times: Closed
• TestAsyncProcess does not pass on HBASE-10070: Closed
• Enable table doesn't balance out replicas evenly if the replicas were unassigned earlier: Closed
• Fix RegionStates.getRegionAssignments to not add duplicate regions: Closed
• Replica map update is problematic in RegionStates: Closed
• Unique keys accounting in MultiThreadedReader is incorrect: Closed
• Add integration test to demonstrate performance improvement: Closed
• multi-get should handle replica location missing from cache: Closed
• LoadTestTool should share the connection and connection pool: Closed
• Add integration test for bulkload with replicas: Closed
• Add some tests on a real cluster for replica: multi master, replication: Closed
• TestRegionRebalancing is failing: Closed
• Use HFileLink in opening region files from secondaries: Closed
• support parallel request cancellation for multi-get: Closed
• HBASE-10070: HMaster can abort with NPE in #rebuildUserRegions: Closed
• TestFromClientSideWithCoprocessor#testGetClosestRowBefore fails due to invalid block size: Closed
• Fixes for scans on a replicated table: Closed
• Timeline Consistent region replicas - Phase 2 design: Closed
• Handle splitting/merging of regions that have region_replication greater than one: Closed
• Fix for metas location cache from HBASE-10785: Closed
• Write flush events to WAL: Closed
• Write region open/close events to WAL: Closed
• Write bulk load COMMIT events to WAL: Closed
• Async WAL replication for region replicas: Closed
• Flush / Compaction handling from secondary region replicas: Closed
• Bulk load handling from secondary region replicas: Closed
• RegionLocations::getRegionLocation can return unexpected replica: Closed
• Add support for doing get/scans against a particular replica_id: Closed
• hbase:meta's regions can be replicated: Closed
• Failover handling for secondary region replicas: Closed
• Integration test for async wal replication to secondary regions: Closed
• Directly invoking split & merge of replica regions should be disallowed: Closed
• Region replicas should be added to the meta table at the time of table creation: Closed
• Improve cancellation for the scan RPCs: Closed
• Maintain SeqId monotonically increasing: Closed
• Replicas of regions can be cached from different instances of the table in MetaCache: Closed
• Handling memory pressure for secondary region replicas: Closed
• RegionReplicaReplicationEndpoint should not set the RPC Codec: Closed
• Disallow non-atomic update operations when TIMELINE consistency is enabled: Closed
• Async wal replication for region replicas and dist log replay does not work together: Closed
• ModifyTable increasing the region replica count should also auto-setup RRRE: Closed
• Split out locality metrics among primary and secondary region: Closed
• Handle FileNotFoundException in region replica replay for flush/compaction events: Closed
• Update documentation for 10070 Phase 2 changes: Closed