[HBASE-20828] Finish-up AMv2 Design/List of Tenets/Specification of operation

• Type: Umbrella
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: amv2
• Assignee: Unassigned
• Reporter: Michael Stack
• Created: 01/Jul/18 01:54
• Updated: 24/Oct/18 00:39

Description

AMv2 is missing specification. There are too many grey-areas still. Also missing are a concise listing of the tenets of AMv2 operation. Here are some examples:

• HBASE-19529 "Handle null states in AM": Asks how we should treat null state in hbase:meta. What does it 'mean'. We seem to treat it differently dependent on context. Needs clarification. Apache9 recently asked similar about the meaning of OFFLINE.
• Logging needs to have a particular form to help trace Procedure progress; needs a write-up.

Lets fill in items to address in this umbrella issue. Can address in subissues and produce specification doc too. We have the below but these are mostly (incomplete) description for devs on pv2 and amv2; the specification is missing:

http://hbase.apache.org/book.html#pv2
http://hbase.apache.org/book.html#amv2

(Other areas include addressing what is up w/ rollback – when, how much, and when it is not appropriate – as well as recommendation on Procedures coarseness, locking – is it ok to lock table in alter table procedure for the life of the procedure? – and so on).

Sub-Tasks

• Handle null states in AM: Resolved
• Restore procedure locks when master restarts: Resolved
• The parent procedure of RegionTransitionProcedure may not have the table lock: Resolved
• Merged region's RIT state may not be cleaned after master restart: Resolved
• RS was killed due to master thought the region should be on a already dead server: Closed
• Data loss if merging regions while ServerCrashProcedure executing: Resolved
• Introduce a region transition procedure to handle all the state transition for a region: Resolved
• RS may get killed while master restarts: Resolved
• Data loss if splitting region while ServerCrashProcedure executing: Resolved
• Merge the update procedure store on locking with the general persist after a procedure execution: Open
• Possible NPE in ReopenTableRegionsProcedure: Resolved
• There will be race when we call suspendIfNotReady and then throw ProcedureSuspendedException: Resolved
• Split/Merge table can be executed concurrently with DisableTableProcedure: Resolved
• ArrayIndexOutOfBoundsException when rolling back procedure: Resolved
• Lock may not be taken or released while rolling back procedure: Resolved
• SCP can be scheduled multiple times for the same RS: Resolved
• One operation in procedure batch throws an exception will cause all RegionTransitionProcedures receive the same exception: Resolved
• Find another way to test the backoff mechanism in TRSP: Resolved
• Revisit the expected states for open/close: Resolved
• Add cache for TableStateManager: Resolved
• Meta Table should be able to online even if all procedures are lost: Resolved
• Procedure worker should not quit when getting unexpected error: Patch Available
• Exclusive lock may be held by a SUCCESS state procedure forever: Resolved
• Possible NPE if ModifyTable and region split happen at the same time: Resolved
• Confirm that we can (rolling) upgrade from 2.0.x and 2.1.x to 2.2.x after HBASE-20881: Resolved
• Reimplement assign/unassign related procedure metrics: Resolved
• Introduce a mechanism to bypass the execution of a stuck procedure: Resolved
• Adding getter methods to some private fields in ProcedureV2 module: Resolved
• Move TestCreateTableProcedure.testMRegions to a separated file: Resolved
• Remove the explicit timeout config for TestTruncateTableProcedure: Resolved
• The timeout retry logic for several procedures are broken after master restarts: Resolved
• ReopenTableRegionsProcedure sometimes hangs: Resolved
• Reimplement the retry backoff logic for ReopenTableRegionsProcedure: Resolved
• Race in region opening and load balancing can cause region stuck in RIT: Resolved
• Revisit the executeProcedure method for open/close region: Resolved
• Revisit the close region related code at RS side: Open
• Implement exponential retrying backoff for Assign/UnassignRegionHandler introduced in HBASE-21217: Resolved
• Allow the procedure implementation to skip persistence of the state after a execution: Resolved
• Rename the closed procedure wal files so that we do not need to call recoverLease when restarting: Resolved
• Load procedure wals with multiple threads: Resolved
• Use CompatRemoteProcedureResolver to dispatch open/close region requests to RS: Resolved
• Skip persistence when retrying for assignment related procedures: Resolved
• Add jitter for ProcedureUtil.getBackoffTimeMs: Resolved
• Refactor WALProcedureStore and add more comments for better understanding the implementation: Resolved
• Need to find a way to limit the number of proc wal files: Resolved
• Add more tests for procedure store related classes: Resolved
• HostingServer in UnassignProcedure is not accurate: Resolved
• Do not rollback successful sub procedures when rolling back a procedure: Resolved
• TransitRegionStateProcedure should not fail with FAILED_OPEN when acting as a sub procedure: Open
• The implementation of BitSetNode is not efficient: Resolved
• The getActiveMinProcId and getActiveMaxProcId of BitSetNode are incorrect if there are no active procedure: Resolved
• Backport HBASE-21278 to branch-2.1 and branch-2.0: Resolved
• Should not skip force updating for a sub procedure even if it has been finished: Resolved
• ReopenTableRegionsProcedure will enter an infinite loop if we schedule a TRSP at the same time: Resolved
• Simplify the implementation of WALProcedureMap: Resolved
• The force update thread may have race with PE worker when the procedure is rolling back: Resolved
• Polish the rollback implementation in ProcedureExecutor: Open
• Procedure may be deleted improperly during master restarts resulting in 'Corrupt': Resolved
• Rewrite the buildingHoldCleanupTracker method in WALProcedureStore: Resolved
• Procedure holds the lock should put to front of the queue after restart: Resolved
• Revisit the lock and queue implementation in MasterProcedureScheduler: Resolved
• Add some verbose log to MasterProcedureScheduler: Resolved
• Add debug log for procedure stack id related operations: Resolved
• Procedure with holdlock=false should not be restored lock when restarts: Resolved
• Abort split/merge procedure if there is a table procedure of the same table going on: Resolved
• Do not kill RS if reportOnlineRegions fails: Resolved
• Procedures for meta table/region should be able to execute in separate workers: Resolved
• The checkOnlineRegionsReport can accidentally complete a TRSP: Resolved
• Retry on reportRegionStateTransition can lead to unexpected errors: Resolved
• Add more comments about how we do fencing for WALProcedureStore: Resolved
• Should not persist the dispatched field for RegionRemoteProcedureBase: Resolved
• Backport "HBASE-21463 The checkOnlineRegionsReport can accidentally complete a TRSP" to branch-2.1 and branch-2.0: Resolved
• WALProcedure may remove proc wal files still with active procedures: Resolved
• Ignore the reportRegionStateTransition call from a dead server: Resolved