Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-20828 Finish-up AMv2 Design/List of Tenets/Specification of operation
  3. HBASE-21423

Procedures for meta table/region should be able to execute in separate workers

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.1, 2.0.2
    • Fix Version/s: 2.0.3, 2.1.2
    • Component/s: None
    • Labels:
      None
    • Release Note:
      Hide
      The procedure for meta table will be executed in a separate worker thread named 'Urgent Worker' to avoid stuck. A new config named 'hbase.master.urgent.procedure.threads' is added, the default value for it is 1. To disable the separate worker, set it to 0.
      Show
      The procedure for meta table will be executed in a separate worker thread named 'Urgent Worker' to avoid stuck. A new config named 'hbase.master.urgent.procedure.threads' is added, the default value for it is 1. To disable the separate worker, set it to 0.

      Description

      We have higher priority for meta table procedures, but only in queue level. There is a case that the meta table is closed and a AssignProcedure(or RTSP in branch-2+) is waiting there to be executed, but at the same time, all the Work threads are executing procedures need to write to meta table, then all the worker will be stuck and retry for writing meta, no worker will take the AP for meta.
      Though we have a mechanism that will detect stuck and adding more ''KeepAlive'' workers to the pool to resolve the stuck. It is already stuck a long time.
      This is a real case I encountered in ITBLL.
      So, I add one 'Urgent work' to the ProceudureExecutor, which only take meta procedures(other workers can take meta procedures too), which can resolve this kind of stuck.

        Attachments

        1. HBASE-21423.branch-2.0.001.patch
          20 kB
          Allan Yang
        2. HBASE-21423.branch-2.0.002.patch
          29 kB
          Allan Yang
        3. HBASE-21423.branch-2.0.003.patch
          30 kB
          Allan Yang
        4. HBASE-21423.branch-2.0.addendum.patch
          16 kB
          Allan Yang

        Issue Links

          Activity

            People

            • Assignee:
              allan163 Allan Yang
              Reporter:
              allan163 Allan Yang

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment