Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.0.0-alpha-1
-
None
Description
The MOB compaction catches FileNotFoundException when hbase.unsafe.mob.discard.miss is true to handle missing MOB cells. The FNFE is wrapped in DoNotRetryIOException so the compaction fails for the given region.
2021-07-21 13:51:05,880 WARN org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor: hbase.unsafe.mob.discard.miss=true. This is unsafe setting recommended only when first upgrading to a version with the distributed mob compaction feature on a cluster that has experienced MOB data corruption. 2021-07-21 13:51:05,880 WARN org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor: hbase.unsafe.mob.discard.miss=true. This is unsafe setting recommended only when first upgrading to a version with the distributed mob compaction feature on a cluster that has experienced MOB data corruption. 2021-07-21 13:51:05,880 INFO org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor: Compact MOB=true optimized configured=false optimized enabled=false maximum MOB file size=1073741824 major=true store=[table=IntegrationTestIngestWithMOB family=test_cf region=3a2ee81f9244c39ba61d694e616c1a89] 2021-07-21 13:51:05,880 INFO org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor: Compact MOB=true optimized configured=false optimized enabled=false maximum MOB file size=1073741824 major=true store=[table=IntegrationTestIngestWithMOB family=test_cf region=7a96f55bb9ae04500a06cbaef02da6a3] 2021-07-21 13:51:05,888 INFO org.apache.hadoop.hbase.regionserver.RSRpcServices: Compacting IntegrationTestIngestWithMOB,77777777,1626787996628.c71cad04514b17ee86a407490bd27424. 2021-07-21 13:51:05,891 INFO org.apache.hadoop.hbase.regionserver.RSRpcServices: Compacting IntegrationTestIngestWithMOB,88888888,1626787996628.8fd002bda07755decda67b7084d1e0f6. 2021-07-21 13:51:05,895 ERROR org.apache.hadoop.hbase.regionserver.HMobStore: The mob file 1bbd886460827015e5d605ed44252251202107200e5065290b424e38992f5556d9943b6a_7a96f55bb9ae04500a06cbaef02da6a3 could not be found in the locations [hdfs://example.com:8020/hbase/mobdir/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce5 3c2/test_cf, hdfs://example.com:8020/hbase/archive/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf] or it is corrupt 2021-07-21 13:51:05,895 INFO org.apache.hadoop.hbase.regionserver.throttle.PressureAwareThroughputController: 7a96f55bb9ae04500a06cbaef02da6a3#test_cf#compaction#1 average throughput is 0.07 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 10.00 MB/second 2021-07-21 13:51:05,908 INFO org.apache.hadoop.hbase.regionserver.RSRpcServices: Compacting IntegrationTestIngestWithMOB,dddddddd,1626787996628.53186ca5008e3a964eee5f96ee3f1b26. 2021-07-21 13:51:05,997 ERROR org.apache.hadoop.hbase.regionserver.CompactSplit: Compaction failed Request=regionName=IntegrationTestIngestWithMOB,11111111,1626787996628.7a96f55bb9ae04500a06cbaef02da6a3., storeName=test_cf, fileCount=1, fileSize=110.6 M (110.6 M), priority=1, time=1626875465819 java.io.IOException: Mob compaction failed for region: 7a96f55bb9ae04500a06cbaef02da6a3 at org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.performCompaction(DefaultMobStoreCompactor.java:575) at org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:327) at org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.compact(DefaultMobStoreCompactor.java:227) at org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:126) at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1407) at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2183) at org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:633) at org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:675) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.io.FileNotFoundException: File does not exist: hdfs://example.com:8020/hbase/archive/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf/1bbd886460827015e5d605ed44252251202107200e5065290b424e38992f5556d9943b6a_7a96f55bb9ae04500a06cbaef02da6a3 at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:438) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:328) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:307) at org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.performCompaction(DefaultMobStoreCompactor.java:366) ... 10 more Caused by: java.io.FileNotFoundException: File does not exist: hdfs://example.com:8020/hbase/archive/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf/1bbd886460827015e5d605ed44252251202107200e5065290b424e38992f5556d9943b6a_7a96f55bb9ae04500a06cbaef02da6a3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1500) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1493) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1508) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:140) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:104) at org.apache.hadoop.hbase.regionserver.HStoreFile.<init>(HStoreFile.java:220) at org.apache.hadoop.hbase.mob.CachedMobFile.create(CachedMobFile.java:49) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:220) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:399) ... 13 more 2021-07-21 13:51:06,013 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction of test_cf in IntegrationTestIngestWithMOB,77777777,1626787996628.c71cad04514b17ee86a407490bd27424.
Attachments
Issue Links
- links to