Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-16095

Add priority to TableDescriptor and priority region open thread pool

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3.0, 0.98.21, 2.0.0
    • Component/s: None
    • Labels:
      None
    • Release Note:
      Hide
      Adds a PRIORITY property to the HTableDescriptor. PRIORITY should be in the same range as the RpcScheduler defines it (HConstants.XXX_QOS).

      Table priorities are only used for region opening for now. There can be other uses later (like RpcScheduling).

      Regions of high priority tables (priority >= than HIGH_QOS) are opened from a different thread pool than the regular region open thread pool. However, table priorities are not used as a global order for region assigning or opening.
      Show
      Adds a PRIORITY property to the HTableDescriptor. PRIORITY should be in the same range as the RpcScheduler defines it (HConstants.XXX_QOS). Table priorities are only used for region opening for now. There can be other uses later (like RpcScheduling). Regions of high priority tables (priority >= than HIGH_QOS) are opened from a different thread pool than the regular region open thread pool. However, table priorities are not used as a global order for region assigning or opening.

      Description

      This is in the similar area with HBASE-15816, and also required with the current secondary indexing for Phoenix.

      The problem with P secondary indexes is that data table regions depend on index regions to be able to make progress. Possible distributed deadlocks can be prevented via custom RpcScheduler + RpcController configuration via HBASE-11048 and PHOENIX-938. However, region opening also has the same deadlock situation, because data region open has to replay the WAL edits to the index regions. There is only 1 thread pool to open regions with 3 workers by default. So if the cluster is recovering / restarting from scratch, the deadlock happens because some index regions cannot be opened due to them being in the same queue waiting for data regions to open (which waits for RPC'ing to index regions which is not open). This is reproduced in almost all Phoenix secondary index clusters (mutable table w/o transactions) that we see.

      The proposal is to have a "high priority" region opening thread pool, and have the HTD carry the relative priority of a table. This maybe useful for other "framework" level tables from Phoenix, Tephra, Trafodian, etc if they want some specific tables to become online faster.

      As a follow up patch, we can also take a look at how this priority information can be used by the rpc scheduler on the server side or rpc controller on the client side, so that we do not have to set priorities manually per-operation.

        Attachments

        1. hbase-16095_v0.patch
          9 kB
          Enis Soztutar
        2. hbase-16095_v1.patch
          15 kB
          Enis Soztutar
        3. hbase-16095_v2.patch
          15 kB
          Enis Soztutar
        4. hbase-16095_v3.patch
          17 kB
          Enis Soztutar
        5. HBASE-16095-0.98.patch
          18 kB
          Andrew Kyle Purtell
        6. HBASE-16095-0.98.patch
          18 kB
          Andrew Kyle Purtell

          Issue Links

            Activity

              People

              • Assignee:
                enis Enis Soztutar
                Reporter:
                enis Enis Soztutar
              • Votes:
                0 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: