[HBASE-21423] Procedures for meta table/region should be able to execute in separate workers

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.1, 2.0.2
• Fix Version/s: 2.0.3, 2.1.2

Release Note

The procedure for meta table will be executed in a separate worker thread named 'Urgent Worker' to avoid stuck. A new config named 'hbase.master.urgent.procedure.threads' is added, the default value for it is 1. To disable the separate worker, set it to 0.

• Assignee: Allan Yang
• Reporter: Allan Yang
• Created: 01/Nov/18 15:13
• Updated: 14/Nov/18 07:03
• Resolved: 12/Nov/18 02:48

Description

We have higher priority for meta table procedures, but only in queue level. There is a case that the meta table is closed and a AssignProcedure(or RTSP in branch-2+) is waiting there to be executed, but at the same time, all the Work threads are executing procedures need to write to meta table, then all the worker will be stuck and retry for writing meta, no worker will take the AP for meta.
Though we have a mechanism that will detect stuck and adding more ''KeepAlive'' workers to the pool to resolve the stuck. It is already stuck a long time.
This is a real case I encountered in ITBLL.
So, I add one 'Urgent work' to the ProceudureExecutor, which only take meta procedures(other workers can take meta procedures too), which can resolve this kind of stuck.

Issue Links

• is related to: HBASE-21468 separate workers for meta table is not working