[HBASE-14614] Procedure v2: Core Assignment Manager

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.0
• Component/s: proc-v2
• Hadoop Flags: Incompatible change, Reviewed

Release Note

Replaces the AssignmentManager with a new procedurev2-based AssignmentManager

h1. AMv2
Puts AssignmentManager up on top of the ProcedureV2 state machine with persistence engine. Each assignment atom is now a Procedure implementation; e.g. an AssignProcedure and an UnassignProcedure. Molecules of aggregated Procedures are used to do more involved assignment steps: e.g. the move region procedure is made of an Unassign followed by an Assign subprocedure.

AMv2 is 1500 lines. Old AM was near 4000. Functionality has been moved out to Procedures. In-memory states of regions and servers has been cleaned up stored in new RegionStates implementation. RegionStateStore takes care of publishing final region state out to the hbase:meta table.

New RemoteProcedureDispatcher/RSProcedureDispatcher runs the Procedure-based assignments ‘remotely’. Knows about ‘servers’. Does aggregation of assignments by time on a time/count basis so can send procedures in batches rather than one per RPC. Procedure status comes back on the back of the RegionServer heartbeat reporting online regions. The response is passed to the AMv2 to ‘process’. It will check against the in-memory state. If there is a mismatch, it fences out the RegionServer on the assumption that something went wrong on the RS side.Timeouts trigger retries. The Procedure machine ensures only one operation at a time on any one region/table using locking and smarts about what is serial and what can be run concurrently.

New accounting of RegionServer version will be used running rolling restarts.

‘States’ -- OPENING, CLOSING, etc. -- are now in-memory in-the-master only serialized out to the ProcedureV2 WAL. They are no longer persisted to ZooKeeper.

h2. Assign Detail
The Assign starts by pushing the "assign" operation to the AssignmentManager and then will go into a “waiting" state. The AM will batch the "assign" requests and ask the Balancer where to put the region (the various policies will be respected: retain, round-robin, random). Once the AM and the balancer have found a place for the region, the procedure will be resumed and an "open region" request will be placed in the Remote Dispatcher queue, and the procedure once again will go into a "waiting state". The Remote Dispatcher will batch the various requests for that server and they will be sent to the RS for execution. The RS will complete the open operation by calling master.reportRegionStateTransition(). The AM will intercept the transition report, and notify the procedure. The procedure will finish the assignment by publishing to new state on hbase:meta or it will retry the assignment.

h3. Unassign Detail
 The Unassign starts by placing a "close region" request in the Remote Dispatcher queue, and the procedure will then go into a "waiting state". The Remote Dispatcher will batch the various requests for that server and they will be sent to the RS for execution. The RS will complete the open operation by calling master.reportRegionStateTransition(). The AM will intercept the transition report, and notify the procedure. The procedure will finish the unassign by publishing its new state on meta or it will retry the unassign.

h1. New Configs
 * "hbase.procedure.remote.dispatcher.threadpool.size" defaults 128
 * "hbase.procedure.remote.dispatcher.delay.msec" default 150ms
 * "hbase.procedure.remote.dispatcher.max.queue.size" with default 32
 * "hbase.regionserver.rpc.startup.waittime" with default 60 seconds.
h1. TODO
As of this writing.

Put up a model diagram.

 * Handle region migration
 * Handle meta assignment first
 * Handle sys table assignment first (e.g. acl, namespace)
 * Handle table priorities
 * Do we report same AM metrics as we used too? We do it all in here now.

INCOMPATIBLE
A known incompatible is that because splits and merges are now run from the master, Coprocessors that used to watch for merge/split from a RegionObserver now no longer work; to watch split/merges, you need to have an observer on the Master instead.

• Assignee: Matteo Bertozzi
• Reporter: Stephen Yuan Jiang
• Created: 15/Oct/15 16:52
• Updated: 17/Jun/22 18:02
• Resolved: 02/Jun/17 23:54

Description

New AssignmentManager implemented using proc-v2.

• AssignProcedure handle assignment operation
• UnassignProcedure handle unassign operation
• MoveRegionProcedure handle move/balance operation

Concurrent Assign operations are batched together and sent to the balancer
Concurrent Assign and Unassign operation ready to be sent to the RS are batched together

This patch is an intermediate state where we add the new AM as AssignmentManager2() to the master, to be reached by tests. but the new AM will not be integrated with the rest of the system. Only new am unit-tests will exercise the new assigment manager. The integration with the master code is part of HBASE-14616

Issue Links

• blocks: HBASE-14615 Procedure V2: Implement single region assignment procedure from Master-side
• blocks: HBASE-14616 Procedure v2: Replace the old AM with the new AM
• incorporates: HBASE-14615 Procedure V2: Implement single region assignment procedure from Master-side
• incorporates: HBASE-17844 Subset of HBASE-14614, Procedure v2: Core Assignment Manager (non-critical related changes)
• relates to: HBASE-25207 Revisit the implementation and usage of RegionStates.include
• links to: AM Doc
• links to: Review Board (master)