[HBASE-22433] Corrupt hfile data

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 2.2.0
• Assignee: Unassigned
• Reporter: Lijin Bin
• Created: 16/May/19 02:48
• Updated: 22/May/19 12:41

Description

We use 2.2.0 version and encounter corrupt cell data.

2019-05-15 22:53:59,354 ERROR [regionserver/hb-mbasedata-14:16020-longCompactions-1557048533421] regionserver.CompactSplit: Compaction failed region=mktdm_id_src,99999990,1557681762973.255e9adde013e370deb595c59a7285c3., storeName=o, priority=196, startTime=1557931927314
java.lang.IllegalStateException: Invalid currKeyLen 1700752997 or currValueLen 2002739568. Block offset: 70452918, block length: 66556, position: 42364 (without header).
 at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.checkKeyValueLen(HFileReaderImpl.java:1182)
 at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.readKeyValueLen(HFileReaderImpl.java:628)
 at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl._next(HFileReaderImpl.java:1080)
 at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.next(HFileReaderImpl.java:1097)
 at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:208)
 at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:120)
 at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:644)
 at org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:386)
 at org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:326)
 at org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:65)
 at org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:126)
 at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1429)
 at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2231)
 at org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:629)
 at org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:671)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)

 

2019-05-15 23:14:24,143 ERROR [regionserver/hb-mbasedata-14:16020-longCompactions-1557048533422] regionserver.CompactSplit: Compaction failed region=mktdm_id_src,9fdee4,1557681762973.1782aebb83eae551e7bdfc2bfa13eb3d., storeName=o, priority=194, startTime=1557932726849
java.lang.RuntimeException: Unknown code 98
 at org.apache.hadoop.hbase.KeyValue$Type.codeToType(KeyValue.java:274)
 at org.apache.hadoop.hbase.CellUtil.getCellKeyAsString(CellUtil.java:1307)
 at org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.getMidpoint(HFileWriterImpl.java:383)
 at org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.finishBlock(HFileWriterImpl.java:343)
 at org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.close(HFileWriterImpl.java:603)
 at org.apache.hadoop.hbase.regionserver.StoreFileWriter.close(StoreFileWriter.java:376)
 at org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.abortWriter(DefaultCompactor.java:98)
 at org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.abortWriter(DefaultCompactor.java:42)
 at org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:335)
 at org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:65)
 at org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:126)
 at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1429)
 at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2231)
 at org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:629)
 at org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:671)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)

Issue Links

• is related to: HBASE-20447 Only fail cacheBlock if block collisions aren't related to next block metadata
• is related to: HBASE-20789 TestBucketCache#testCacheBlockNextBlockMetadataMissing is flaky
• is related to: HBASE-22441 BucketCache NullPointerException in cacheBlock
• relates to: HBASE-22445 Add file info when throw exceptions in HFileReaderImpl

Sub-Tasks

• Check refCount before free block in BucketCache: Resolved
• Add file info when throw exceptions in HFileReaderImpl: Resolved
• Get block from BlockCache once and return this block to BlockCache twice make ref count error.: Resolved