[HBASE-14350] Procedure V2 Phase 2: Assignment Manager

• Type: Task
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.0-beta-1, 2.0.0
• Component/s: master, proc-v2

Release Note

(Incomplete)

= Incompatbiles

== Coprocessor Incompatibilities

Split/Merge have moved to the Master; it runs them now. Means hooks around Split/Merge are now noops. To intercept Split/Merge phases, CPs need to intercept on MasterObserver.

• Assignee: Stephen Yuan Jiang
• Reporter: Stephen Yuan Jiang
• Created: 01/Sep/15 06:07
• Updated: 17/Jun/22 18:00
• Resolved: 09/Nov/17 22:07

Description

This is the second phase of Procedure V2 (HBASE-12439). Built on top of Phase 1 (HBASE-14336), implement a new Assignment Manager based on Proc-V2.

Issue Links

• blocks: HBASE-16537 Add tests to verify create/modify table region reopen with missing coprocessor
• Is contained by: HBASE-12439 Procedure V2
• relates to: HBASE-10025 Region stuck in transition after failed RS_ZK_REGION_FAILED_OPEN
• links to: AMv2 for Devs
• links to: AMv2 Overview
• links to: Setup for Phase 2, the Assignment Manager

Sub-Tasks

• Procedure v2: Core Assignment Manager: Closed
• Procedure v2: Replace the old AM with the new AM: Closed
• Procedure V2: Implement move shell command to use Proc-V2 assignment: Closed
• Procedure V2: unnecessaery operations on AssignmentManager#recoverTableInDisablingState() and recoverTableInEnablingState(): Closed
• Procedure V2: Implement single region assignment procedure from Master-side: Closed
• Procedure V2: Update ServerCrashProcedure to interact with assignment procedures: Closed
• Remove unused LoadBalancer.immediateAssignment(): Closed
• Procedure V2 - Completed support parent-child procedure: Closed
• Replace master.am.getTableStateManager() with the direct master.getTableStateManager(): Closed
• replace master.am and master.sm direct access with getter calls: Closed
• Remove HMaster.assignRegion(): Closed
• Procedure V2 - Procedure Queue with Regions: Closed
• Procedure v2 - Improve cleaning up of proc wals: Closed
• Procedure v2 - Reimplement split: Closed
• Procedure v2 - Reimplement merge: Closed
• Require only MasterServices to the ServerManager constructor: Closed
• Procedure V2: Reimplement DispatchMergingRegionHandler: Closed
• Move meta bootstrap out of HMaster: Closed
• Procedure V2: Support acquire/release shared table lock concurrently: Closed
• Procedure V2 - Add support to addChildProcedure() as last "step" in StateMachineProcedure: Closed
• Separate Create/Modify Table operations from open/reopen regions: Closed
• Procedure v2 - Add handling of split region transition to the new AM: Closed
• Procedure v2 - Add new AM metrics: Closed
• Procedure V2 - Recover 'updated' part of WAL tracker if trailer is corrupted.: Closed
• Procedure v2 - Compute WALs cleanup on wal modification and not on every sync: Closed
• Procedure v2 - Support for parent holding locks: Closed
• Procedure V2 - Lock procedures to allow clients to acquire locks on tables/namespaces/regions: Closed
• Procedure V2 - Move ZK-lock's uses to Procedure framework locks (LockProcedure): Closed
• Procedure V2 - Remove org.apache.hadoop.hbase.zookeeper.lock: Closed
• Procedure v2 - Fix StateMachineProcedure support for child procs at last step: Closed
• Procedure v2 - Inherit lock from root proc: Closed
• Procedure V2 - Check ACLs for remote HBaseLock: Closed
• Procedure v2 - add waiting procs back to the queue after restart: Closed
• Procedure v2 - A couple of tweaks to the SplitTableRegionProcedure: Closed
• Procedure v2 - A couple of tweaks to the SplitTableRegionProcedure: Closed
• Procedure v2 - remove tryAcquire*Lock and use wait/wake to make framework event based: Closed
• Procedure v2 - inherit region locks: Closed
• Remove merge region code from region server: Closed
• Remove split region code from Region Server: Closed
• Procedure v2 - cleanup isSuspended from MasterProcedureScheduler#Queue: Closed
• Procedure v2 - Add handling of merge region transition to the new AM: Closed
• Readd TestMergeTableRegionsProcedure mistakenly removed by HBASE-16786: Closed
• Tooling to move table out of FAILED_OPEN state: Closed
• Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload): Closed
• Subset of HBASE-14614, Procedure v2: Core Assignment Manager (non-critical related changes): Closed
• Implement abort for TruncateTableProcedure: Closed
• Support abort for all procedures by default: Closed
• Fix unit tests for TestTableFavoredNodes: Closed
• Add API for who currently holds a lock on namespace/ table/ region and log when state is LOCK_EVENT_WAIT: Closed
• [SHELL] Purge close_region command that allows by-pass of Master: Closed
• [AMv2] If Master gives OPEN to another, if original eventually succeeds, Master will kill it: Closed
• [AMv2] Enable aggregation of RPCs (assigns/unassigns, etc.): Closed
• [AMv2] Split/Merge need cleanup; currently they diverge and do not fully embrace AMv2 world: Closed
• Redo ProcedureInfo and LockInfo: Closed
• [AMv2] Remove DispatchMergingRegionsRequest & DispatchMergingRegions: Closed
• Procedure WALs are archived but not cleaned; fix: Closed
• Assign system tables first (priority): Closed
• Verify Master carrying system-tables only with short-circuit RPC: Closed
• create new Async Split API to embrace AM v2: Closed
• Deprecate and throw unsupported operation when Admin#closeRegion is called.: Closed
• Reduce ProcedureInfo usage: Closed
• Add pv2 and amv2 chapters to refguide: Closed