[HBASE-21083] Introduce a mechanism to bypass the execution of a stuck procedure

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0, 2.0.1
• Fix Version/s: 3.0.0-alpha-1, 2.2.0, 2.1.1, 2.0.2
• Component/s: amv2
• Hadoop Flags: Reviewed
• Assignee: Allan Yang
• Reporter: Allan Yang
• Created: 21/Aug/18 08:58
• Updated: 04/Mar/19 09:05
• Resolved: 29/Aug/18 03:22

Description

Offline discussed with stack and Apache9. We all agreed that we need to introduce a mechanism to 'force complete' a stuck procedure, so the AMv2 can continue running.
we still have some unrevealed bugs hiding in our AMv2 and procedureV2 system, we need something to interfere with stuck procedures before HBCK2 can work. This is very crucial for a production ready system.

For now, we have little ways to interfere with running procedures. Aborting them is not a good choice, since some procedures are not abort-able. And some procedure may have overridden the abort() method, which will ignore the abort request.

So, here, I will introduce a mechanism to bypass the execution of a stuck procedure.
Basically, I added a field called 'bypass' to Procedure class. If we set this field to true, all the logic in execute/rollback will be skipped, letting this procedure and its ancestors complete normally and releasing the lock resources at last.

Notice that bypassing a procedure may leave the cluster in a middle state, e.g. the region not assigned, or some hdfs files left behind.
The Operators need know the side effect of bypassing and recover the inconsistent state of the cluster themselves, like issuing new procedures to assign the regions.

A patch will be uploaded and review board will be open. For now, only APIs in ProcedureExecutor are provided. If anything is fine, I will add it to master service and add a shell command to bypass a procedure. Or, maybe we can use dynamically compiled JSPs to execute those APIs as mentioned in HBASE-20679.

Issue Links

• is related to: HBASE-21023 Add bypassProcedureToCompletion() API to HbckService
• is related to: HBASE-19121 HBCK for AMv2 (A.K.A HBCK2)
• links to: Review Board
• links to: Review Board (branch-2.1)