Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-25206

Data loss can happen if a cloned table loses original split region(delete table)

    XMLWordPrintableJSON

    Details

    • Hadoop Flags:
      Reviewed

      Description

      Steps to reproduce are as follows:

      1. Create a table and put some data into the table:

      create 'test1','cf'
      put 'test1','r1','cf','v1'
      put 'test1','r2','cf','v2'
      put 'test1','r3','cf','v3'
      put 'test1','r4','cf','v4'
      put 'test1','r5','cf','v5'
      

      2. Take a snapshot for the table:

      snapshot 'test1','snap_test'
      

      3. Clone the snapshot to another table

      clone_snapshot 'snap_test','test2'
      

      4. Split the original table

      split 'test1','r3'
      

      5. Drop the original table

      disable 'test1'
      drop 'test1'
      

      After that, we see the error like the following in RS log when opening the regions of the cloned table:

      2020-10-20 13:32:18,415 WARN org.apache.hadoop.hbase.regionserver.HRegion: Failed initialize of region= test2,,1603200595702.bebdc4f740626206eeccad96b7643261., starting to roll back memstore
      java.io.IOException: java.io.IOException: java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://<NN HOST>:8020/hbase/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/.tmp/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/mobdir/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/archive/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89]
              at org.apache.hadoop.hbase.regionserver.HRegion.initializeStores(HRegion.java:1095)
              at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:943)
              at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:899)
              at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7246)
              at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7204)
              at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7176)
              at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7134)
              at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7085)
              at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:283)
              at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
              at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      Caused by: java.io.IOException: java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://<NN HOST>:8020/hbase/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/.tmp/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/mobdir/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/archive/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89]
              at org.apache.hadoop.hbase.regionserver.HStore.openStoreFiles(HStore.java:590)
              at org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:557)
              at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:303)
              at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:5731)
              at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1059)
              at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1056)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              ... 3 more
      Caused by: java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://<NN HOST>:8020/hbase/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/.tmp/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/mobdir/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/archive/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89]
              at org.apache.hadoop.hbase.io.FileLink$FileLinkInputStream.tryOpen(FileLink.java:322)
              at org.apache.hadoop.hbase.io.FileLink$FileLinkInputStream.<init>(FileLink.java:126)
              at org.apache.hadoop.hbase.io.FileLink$FileLinkInputStream.<init>(FileLink.java:117)
              at org.apache.hadoop.hbase.io.FileLink.open(FileLink.java:429)
              at org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.<init>(FSDataInputStreamWrapper.java:125)
              at org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.<init>(FSDataInputStreamWrapper.java:107)
              at org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:261)
              at org.apache.hadoop.hbase.regionserver.HStoreFile.open(HStoreFile.java:368)
              at org.apache.hadoop.hbase.regionserver.HStoreFile.initReader(HStoreFile.java:476)
              at org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:703)
              at org.apache.hadoop.hbase.regionserver.HStore.lambda$openStoreFiles$1(HStore.java:573)
              ... 6 more
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                brfrn169 Toshihiro Suzuki
                Reporter:
                brfrn169 Toshihiro Suzuki
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: