1. HBase
  2. HBASE-4335

Splits can create temporary holes in .META. that confuse clients and regionservers


    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.90.4
    • Fix Version/s: 0.92.0
    • Component/s: regionserver
    • Labels:
    • Hadoop Flags:


      When a SplitTransaction is performed, three updates are done to .META.:
      1. The parent region is marked as splitting (and hence offline)
      2. The first daughter region is added (same start key as parent)
      3. The second daughter region is added (split key is start key)
      (later, the original parent region is deleted, but that's not important to this discussion)

      Steps 2 and 3 are actually done concurrently by SplitTransaction.DaughterOpener threads. While the master is notified when a split is complete, the only visibility that clients have is whether the daughter regions have appeared in .META.

      If the second daughter is added to .META. first, then .META. will contain the (offline) parent region followed by the second daughter region. If the client looks up a key that is greater than (or equal to) the split, the client will find the second daughter region and use it. If the key is less than the split key, the client will find the parent region and see that it is offline, triggering a retry.

      If the first daughter is added to .META. before the second daughter, there is a window during which .META. has a hole: the first daughter effectively hides the parent region (same start key), but there is no entry for the second daughter. A region lookup will find the first daughter for all keys in the parent's range, but the first daughter does not include keys at or beyond the split key.

      See HBASE-4333 and HBASE-4334 for details on how this causes problems and suggestions for mitigating this in the client and regionserver.

      1. 4335-v5.txt
        15 kB
        Lars Hofhansl
      2. 4335-v4.txt
        14 kB
        Lars Hofhansl
      3. 4335-v3.txt
        15 kB
        Lars Hofhansl
      4. 4335-v2.txt
        4 kB
        Lars Hofhansl
      5. 4335.txt
        4 kB
        Lars Hofhansl

        Issue Links


          Joe Pallas created issue -
          Joe Pallas made changes -
          Field Original Value New Value
          Link This issue is related to HBASE-2600 [ HBASE-2600 ]
          stack made changes -
          Fix Version/s 0.92.0 [ 12314223 ]
          Priority Major [ 3 ] Critical [ 2 ]
          Lars Hofhansl made changes -
          Attachment 4335.txt [ 12496373 ]
          Lars Hofhansl made changes -
          Assignee Lars Hofhansl [ lhofhansl ]
          Lars Hofhansl made changes -
          Attachment 4335-v2.txt [ 12496400 ]
          Jonathan Hsieh made changes -
          Link This issue relates to HBASE-4377 [ HBASE-4377 ]
          Lars Hofhansl made changes -
          Attachment 4335-v3.txt [ 12498117 ]
          Lars Hofhansl made changes -
          Attachment 4335-v4.txt [ 12498273 ]
          Lars Hofhansl made changes -
          Attachment 4335-v5.txt [ 12498383 ]
          Lars Hofhansl made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Resolution Fixed [ 1 ]
          Jonathan Hsieh made changes -
          Link This issue depends upon HBASE-3559 [ HBASE-3559 ]
          Jonathan Hsieh made changes -
          Link This issue depends upon HBASE-3789 [ HBASE-3789 ]
          Lars Francke made changes -
          Status Resolved [ 5 ] Closed [ 6 ]


            • Assignee:
              Lars Hofhansl
              Joe Pallas
            • Votes:
              0 Vote for this issue
              4 Start watching this issue


              • Created: