Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-12791

HBase does not attempt to clean up an aborted split when the regionserver shutting down

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.98.0
    • Fix Version/s: 1.0.0, 0.98.10, 1.1.0
    • Component/s: regionserver
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HBase not cleaning the daughter region directories from HDFS if region server shut down after creating the daughter region directories during the split.

      Here the logs.

      -> RS shutdown after creating the daughter regions.

      2014-12-31 09:05:41,406 DEBUG [regionserver60020-splits-1419996941385] zookeeper.ZKAssign: regionserver:60020-0x14a9701e53100d1, quorum=localhost:2181, baseZNode=/hbase Transitioned node 80c665138d4fa32da4d792d8ed13206f from RS_ZK_REQUEST_REGION_SPLIT to RS_ZK_REQUEST_REGION_SPLIT
      2014-12-31 09:05:41,514 DEBUG [regionserver60020-splits-1419996941385] regionserver.HRegion: Closing t,,1419996880699.80c665138d4fa32da4d792d8ed13206f.: disabling compactions & flushes
      2014-12-31 09:05:41,514 DEBUG [regionserver60020-splits-1419996941385] regionserver.HRegion: Updates disabled for region t,,1419996880699.80c665138d4fa32da4d792d8ed13206f.
      2014-12-31 09:05:41,516 INFO  [StoreCloserThread-t,,1419996880699.80c665138d4fa32da4d792d8ed13206f.-1] regionserver.HStore: Closed f
      2014-12-31 09:05:41,518 INFO  [regionserver60020-splits-1419996941385] regionserver.HRegion: Closed t,,1419996880699.80c665138d4fa32da4d792d8ed13206f.
      2014-12-31 09:05:49,922 DEBUG [regionserver60020-splits-1419996941385] regionserver.MetricsRegionSourceImpl: Creating new MetricsRegionSourceImpl for table t dd9731ee43b104da565257ca1539aa8c
      2014-12-31 09:05:49,922 DEBUG [regionserver60020-splits-1419996941385] regionserver.HRegion: Instantiated t,,1419996941401.dd9731ee43b104da565257ca1539aa8c.
      2014-12-31 09:05:49,929 DEBUG [regionserver60020-splits-1419996941385] regionserver.MetricsRegionSourceImpl: Creating new MetricsRegionSourceImpl for table t 2e40a44511c0e187d357d651f13a1dab
      2014-12-31 09:05:49,929 DEBUG [regionserver60020-splits-1419996941385] regionserver.HRegion: Instantiated t,row2,1419996941401.2e40a44511c0e187d357d651f13a1dab.
      Wed Dec 31 09:06:30 IST 2014 Terminating regionserver
      2014-12-31 09:06:30,465 INFO  [Thread-8] regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@42d2282e
      

      -> Skipping rollback if RS stopped or stopping so we end up in dirty daughter regions in HDFS.

      2014-12-31 09:07:49,547 INFO  [regionserver60020-splits-1419996941385] regionserver.SplitRequest: Skip rollback/cleanup of failed split of t,,1419996880699.80c665138d4fa32da4d792d8ed13206f. because server is stopped
      java.io.InterruptedIOException: Interrupted after 0 tries  on 350
              at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:156)
      

      Because of this hbck always showing inconsistencies.

      ERROR: Region { meta => null, hdfs => hdfs://localhost:9000/hbase/data/default/t/2e40a44511c0e187d357d651f13a1dab, deployed =>  } on HDFS, but not listed in hbase:meta or deployed on any region server
      ERROR: Region { meta => null, hdfs => hdfs://localhost:9000/hbase/data/default/t/dd9731ee43b104da565257ca1539aa8c, deployed =>  } on HDFS, but not listed in hbase:meta or deployed on any region server
      

      If we try to repair then we end up in overlap regions in hbase:meta. and both daughter regions and parent are online.

        Attachments

        1. HBASE-12791.patch
          13 kB
          Rajeshbabu Chintaguntla
        2. HBASE-12791_v6.patch
          14 kB
          Rajeshbabu Chintaguntla
        3. HBASE-12791_v6.patch
          14 kB
          Rajeshbabu Chintaguntla
        4. HBASE-12791_v5.patch
          13 kB
          Rajeshbabu Chintaguntla
        5. HBASE-12791_v4.patch
          13 kB
          Rajeshbabu Chintaguntla
        6. HBASE-12791_v4.patch
          13 kB
          Rajeshbabu Chintaguntla
        7. HBASE-12791_v3.patch
          12 kB
          Rajeshbabu Chintaguntla
        8. HBASE-12791_v2.patch
          13 kB
          Rajeshbabu Chintaguntla
        9. HBASE-12791_branch1.patch
          11 kB
          Rajeshbabu Chintaguntla
        10. HBASE-12791_branch1_v3.patch
          14 kB
          rajeshbabu
        11. HBASE-12791_branch1_v2.patch
          13 kB
          Rajeshbabu Chintaguntla
        12. HBASE-12791_addendum.patch
          0.8 kB
          Rajeshbabu Chintaguntla
        13. HBASE-12791_98.patch
          12 kB
          Rajeshbabu Chintaguntla
        14. HBASE-12791_98_v3.patch
          13 kB
          rajeshbabu
        15. HBASE-12791_98_v2.patch
          12 kB
          Rajeshbabu Chintaguntla

          Issue Links

            Activity

              People

              • Assignee:
                rajeshbabu Rajeshbabu Chintaguntla
                Reporter:
                rajeshbabu Rajeshbabu Chintaguntla
              • Votes:
                0 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: