[HBASE-25206] Data loss can happen if a cloned table loses original split region(delete table)

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, 2.3.3, 2.4.0, 2.2.7
• Component/s: proc-v2, Region Assignment, snapshots
• Hadoop Flags: Reviewed
• Assignee: Toshihiro Suzuki
• Reporter: Toshihiro Suzuki
• Created: 20/Oct/20 13:18
• Updated: 17/Jun/22 18:04
• Resolved: 24/Oct/20 15:22

Description

Steps to reproduce are as follows:

1. Create a table and put some data into the table:

create 'test1','cf'
put 'test1','r1','cf','v1'
put 'test1','r2','cf','v2'
put 'test1','r3','cf','v3'
put 'test1','r4','cf','v4'
put 'test1','r5','cf','v5'

2. Take a snapshot for the table:

snapshot 'test1','snap_test'

3. Clone the snapshot to another table

clone_snapshot 'snap_test','test2'

4. Split the original table

split 'test1','r3'

5. Drop the original table

disable 'test1'
drop 'test1'

After that, we see the error like the following in RS log when opening the regions of the cloned table:

2020-10-20 13:32:18,415 WARN org.apache.hadoop.hbase.regionserver.HRegion: Failed initialize of region= test2,,1603200595702.bebdc4f740626206eeccad96b7643261., starting to roll back memstore
java.io.IOException: java.io.IOException: java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://<NN HOST>:8020/hbase/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/.tmp/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/mobdir/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/archive/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89]
        at org.apache.hadoop.hbase.regionserver.HRegion.initializeStores(HRegion.java:1095)
        at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:943)
        at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:899)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7246)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7204)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7176)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7134)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7085)
        at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:283)
        at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://<NN HOST>:8020/hbase/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/.tmp/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/mobdir/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/archive/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89]
        at org.apache.hadoop.hbase.regionserver.HStore.openStoreFiles(HStore.java:590)
        at org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:557)
        at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:303)
        at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:5731)
        at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1059)
        at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1056)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        ... 3 more
Caused by: java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://<NN HOST>:8020/hbase/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/.tmp/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/mobdir/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/archive/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89]
        at org.apache.hadoop.hbase.io.FileLink$FileLinkInputStream.tryOpen(FileLink.java:322)
        at org.apache.hadoop.hbase.io.FileLink$FileLinkInputStream.<init>(FileLink.java:126)
        at org.apache.hadoop.hbase.io.FileLink$FileLinkInputStream.<init>(FileLink.java:117)
        at org.apache.hadoop.hbase.io.FileLink.open(FileLink.java:429)
        at org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.<init>(FSDataInputStreamWrapper.java:125)
        at org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.<init>(FSDataInputStreamWrapper.java:107)
        at org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:261)
        at org.apache.hadoop.hbase.regionserver.HStoreFile.open(HStoreFile.java:368)
        at org.apache.hadoop.hbase.regionserver.HStoreFile.initReader(HStoreFile.java:476)
        at org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:703)
        at org.apache.hadoop.hbase.regionserver.HStore.lambda$openStoreFiles$1(HStore.java:573)
        ... 6 more

Issue Links

• is blocked by: HBASE-25207 Revisit the implementation and usage of RegionStates.include
• links to: GitHub Pull Request #2569