HBase
  1. HBase
  2. HBASE-3789

Cleanup the locking contention in the master

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.90.2
    • Fix Version/s: 0.92.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      The master now creates the znode when closing a region, which is an incompatible change.
      SplitTransaction now waits on the master to delete the znode that it created before it can finish.
      The master doesn't keep track of znodes being deleted and created anymore, it was getting out of sync too easily.
      Show
      The master now creates the znode when closing a region, which is an incompatible change. SplitTransaction now waits on the master to delete the znode that it created before it can finish. The master doesn't keep track of znodes being deleted and created anymore, it was getting out of sync too easily.

      Description

      The new master uses a lot of synchronized blocks to be safe, but it only takes a few jstacks to see that there's multiple layers of lock contention when a bunch of regions are moving (like when the balancer runs). The main culprits are regionInTransition in AssignmentManager, ZKAssign that uses ZKW.getZNnodes (basically another set of region in transitions), and locking at the RegionState level.

      My understanding is that even tho we have multiple threads to handle regions in transition, everything is actually serialized. Most of the time, lock holders are talking to ZK or a region server, which can take a few milliseconds.

      A simple example is when AssignmentManager wants to update the timers for all the regions on a RS, it will usually be waiting on another thread that's holding the lock while talking to ZK.

      1. HBASE-3789-v4-0.90.patch
        27 kB
        Jean-Daniel Cryans
      2. HBASE-3789-trunk.patch
        36 kB
        Jean-Daniel Cryans

        Issue Links

          Activity

          Jean-Daniel Cryans created issue -
          Jean-Daniel Cryans made changes -
          Field Original Value New Value
          Attachment HBASE-3789.patch [ 12476496 ]
          Jean-Daniel Cryans made changes -
          Assignee Jean-Daniel Cryans [ jdcryans ]
          Jean-Daniel Cryans made changes -
          Attachment HBASE-3789-v2-0.90.patch [ 12480610 ]
          Jean-Daniel Cryans made changes -
          Attachment HBASE-3789-v3-0.90.patch [ 12480707 ]
          Jean-Daniel Cryans made changes -
          Attachment HBASE-3789-v3-0.90.patch [ 12480707 ]
          Jean-Daniel Cryans made changes -
          Attachment HBASE-3789-v3-0.90.patch [ 12480708 ]
          Jean-Daniel Cryans made changes -
          Attachment HBASE-3789-trunk-wip.patch [ 12481275 ]
          Jean-Daniel Cryans made changes -
          Attachment HBASE-3789-trunk.patch [ 12481291 ]
          Jean-Daniel Cryans made changes -
          Attachment HBASE-3789-trunk-wip.patch [ 12481275 ]
          Jean-Daniel Cryans made changes -
          Attachment HBASE-3789-v2-0.90.patch [ 12480610 ]
          Jean-Daniel Cryans made changes -
          Attachment HBASE-3789.patch [ 12476496 ]
          Jean-Daniel Cryans made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Hadoop Flags [Incompatible change, Reviewed]
          Release Note The master now creates the znode when closing a region, which is an incompatible change.
          SplitTransaction now waits on the master to delete the znode that it created before it can finish.
          The master doesn't keep track of znodes being deleted and created anymore, it was getting out of sync too easily.
          Resolution Fixed [ 1 ]
          Jean-Daniel Cryans made changes -
          Attachment HBASE-3789-v3-0.90.patch [ 12480708 ]
          Jean-Daniel Cryans made changes -
          Attachment HBASE-3789-v4-0.90.patch [ 12482830 ]
          Jonathan Hsieh made changes -
          Link This issue is depended upon by HBASE-4335 [ HBASE-4335 ]

            People

            • Assignee:
              Jean-Daniel Cryans
              Reporter:
              Jean-Daniel Cryans
            • Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development