[HBASE-20610] Procedure V2 - Distributed Log Splitting

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, 2.2.0
• Component/s: proc-v2
• Hadoop Flags: Reviewed
• Release Note: See RN in HBASE-21588 for detail on this feature. It landed in hbase-2.2.0.
• Assignee: Jingyun Tian
• Reporter: Guanghao Zhang
• Created: 22/May/18 03:48
• Updated: 25/Jun/20 19:08
• Resolved: 17/Jun/20 15:46

Description

Now master and regionserver use zk to coordinate log split tasks. The split log manager manages all log files which need to be scanned and split. Then the split log manager places all the logs into the ZooKeeper splitWAL node (/hbase/splitWAL) as tasks and monitors these task nodes and waits for them to be processed. Each regionserver watch splitWAL znode and grab task when node children changed. And regionserver does the work to split the logs.

Open this umbrella issue to move this "coordinate" work to use new procedure v2 framework and reduce zk depencency. Plan to finish this before 3.0 release. Any suggestions are welcomed. Thanks.

Issue Links

• links to: Design doc
• links to: Review Board (master)

Sub-Tasks

• Procedure v2 wal splitting implementation: Resolved
• Add status track for splitting WAL tasks: Resolved
• Extract ProcedureCoordinatorRpcs and ProcedureMemberRpcs from CoordinatedStateManager: Resolved
• Update HBase-book with the procedure based WAL splitting: Resolved
• RemoteProcedureDispatcher should track the ongoing dispatched calls: Resolved
• Procedure V2 - Distributed WAL Splitting => LOGGING: Resolved
• Enable procedure-based log splitting as default in hbase3: Resolved