Uploaded image for project: 'Phoenix'
  1. Phoenix
  2. PHOENIX-3072

Deadlock on region opening with secondary index recovery

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.9.0, 4.8.1
    • Labels:
      None

      Description

      There is a distributed deadlock happening in clusters with some moderate number of regions for the data tables and secondary index tables and cluster and it is cluster restart or some large failure. We have seen this in a couple of production cases already.

      Opening of regions in hbase is performed by a thread pool with 3 threads by default. Every regionserver can open 3 regions at a time. However, opening data table regions has to write to multiple index regions during WAL recovery. All other region open requests are queued up in a single queue. This causes a deadlock, since the secondary index regions are also opened by the same thread pools that we do the work. So if there is greater number of data table regions then available number of region opening threads from regionservers, the secondary index region open requests just wait to be processed in the queue. Since these index regions are not open, the region opening of data table regions just block the region opening threads for a long time.

      One proposed fix is to use a different thread pool for opening regions of the secondary index tables so that we will not deadlock. See HBASE-16095 for the HBase-level fix. In Phoenix, we just have to set the priority for secondary index tables.

        Attachments

        1. phoenix-3072_v1.patch
          59 kB
          Enis Soztutar
        2. phoenix-3072_v2.patch
          8 kB
          Enis Soztutar
        3. PHOENIX-3072_v3.patch
          12 kB
          James R. Taylor
        4. PHOENIX-3072_v4.patch
          306 kB
          James R. Taylor

          Issue Links

            Activity

              People

              • Assignee:
                enis Enis Soztutar
                Reporter:
                enis Enis Soztutar
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: