[HBASE-25970] MOB data loss - incorrect concatenation of MOB_FILE_REFS

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha-1
• Fix Version/s: 3.0.0-alpha-1, 2.5.0
• Component/s: mob
• Assignee: Peter Somogyi
• Reporter: Peter Somogyi
• Created: 04/Jun/21 14:09
• Updated: 20/Aug/22 20:12
• Resolved: 19/Aug/22 10:54

Description

Active MOB files can be deleted by MobFileCleanerChore. The MOB_FILE_REFS are wrongly concatenated when multiple files or tables are stored. During the MOB cleanup HBase wants to keep the incorrectly concatenated file and considers the real file as deletable.

2021-06-03 15:03:09,876 TRACE org.apache.hadoop.hbase.mob.MobFileCleanerChore: Specific mob references found for store=hdfs://c1377-node2.coelab.cloudera.com:8020/hbase/data/default/IntegrationTestIngestWithMOB/1df6561244ecc434f505501e6bda9af0/test_cf/2956445a41184488b914e93199df905a : {IntegrationTestIngestWithMOB=[d41d8cd98f00b204e9800998ecf8427e20210603a6c8c0e5bf8548649b0009b156452c21_1df6561244ecc434f505501e6bda9af0d41d8cd98f00b204e9800998ecf8427e20210603497714eabbe5418c9d52382a4808e986_1df6561244ecc434f505501e6bda9af0, d41d8cd98f00b204e9800998ecf8427e202106037464cbddc86943a1bad77d52a3bd412c_1df6561244ecc434f505501e6bda9af0]} 
...
2021-06-03 15:03:10,208 TRACE org.apache.hadoop.hbase.mob.MobFileCleanerChore: Archiving MOB file hdfs://c1377-node2.coelab.cloudera.com:8020/hbase/mobdir/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf/d41d8cd98f00b204e9800998ecf8427e20210603497714eabbe5418c9d52382a4808e986_1df6561244ecc434f505501e6bda9af0 creation time=1622729860319
2021-06-03 15:03:10,467 DEBUG org.apache.hadoop.hbase.mob.MobFileCleanerChore: MOB Cleaner is archiving: hdfs://c1377-node2.coelab.cloudera.com:8020/hbase/mobdir/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf/d41d8cd98f00b204e9800998ecf8427e20210603497714eabbe5418c9d52382a4808e986_1df6561244ecc434f505501e6bda9af0

Issue Links

• links to: GitHub Pull Request #3355