Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-20724 Sometimes some compacted storefiles are still opened after region failover
  3. HBASE-22330

Backport HBASE-20724 (Sometimes some compacted storefiles are still opened after region failover) to branch-1

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments


    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.5.0, 1.4.9, 1.3.4
    • Fix Version/s: 1.4.10, 1.3.5
    • Component/s: Compaction, regionserver
    • Labels:
    • Hadoop Flags:


      There appears to be a race condition between close and split which when combined with a side effect of HBASE-20704, leads to the parent region store files getting archived and cleared while daughter regions still have references to those parent region store files.

      Here is the timeline of events observed for an affected region:

      1. RS1 faces ZooKeeper connectivity issue for master node and starts shutting itself down. As part of this it starts to close the store and clean up the compacted files (File A)
      2. Master starts bulk assigning regions and assign parent region to RS2
      3. Region opens on RS2 and ends up opening compacted store file(s) (suspect this is due to HBASE-20724)
      4. Now split happens and daughter regions open on RS2 and try to run a compaction as part of post open
      5. Split request at this point is complete. However now archiving proceeds on RS1 and ends up archiving the store file that is referenced by the daughter.

      Compaction fails due to FileNotFoundException and all subsequent attempts to open the region will fail until manual resolution.

      We think having HBASE-20724 would help in such situations since we won't end up loading compacted store files in the first place. 


        1. HBASE-22330-addendum.branch-1.patch
          2 kB
          Abhishek Singh Chouhan
        2. HBASE-22330.branch-1.3.001.patch
          62 kB
          Abhishek Singh Chouhan
        3. HBASE-22330.branch-1.002.patch
          62 kB
          Abhishek Singh Chouhan
        4. HBASE-22330.branch-1.001.patch
          61 kB
          Abhishek Singh Chouhan



            • Assignee:
              abhishek.chouhan Abhishek Singh Chouhan
              apurtell Andrew Kyle Purtell


              • Created:

                Issue deployment