Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-22433

Corrupt hfile data

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 2.2.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      We use 2.2.0 version and encounter corrupt cell data.

      
      2019-05-15 22:53:59,354 ERROR [regionserver/hb-mbasedata-14:16020-longCompactions-1557048533421] regionserver.CompactSplit: Compaction failed region=mktdm_id_src,99999990,1557681762973.255e9adde013e370deb595c59a7285c3., storeName=o, priority=196, startTime=1557931927314
      java.lang.IllegalStateException: Invalid currKeyLen 1700752997 or currValueLen 2002739568. Block offset: 70452918, block length: 66556, position: 42364 (without header).
       at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.checkKeyValueLen(HFileReaderImpl.java:1182)
       at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.readKeyValueLen(HFileReaderImpl.java:628)
       at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl._next(HFileReaderImpl.java:1080)
       at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.next(HFileReaderImpl.java:1097)
       at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:208)
       at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:120)
       at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:644)
       at org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:386)
       at org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:326)
       at org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:65)
       at org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:126)
       at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1429)
       at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2231)
       at org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:629)
       at org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:671)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)
      
       
      
      2019-05-15 23:14:24,143 ERROR [regionserver/hb-mbasedata-14:16020-longCompactions-1557048533422] regionserver.CompactSplit: Compaction failed region=mktdm_id_src,9fdee4,1557681762973.1782aebb83eae551e7bdfc2bfa13eb3d., storeName=o, priority=194, startTime=1557932726849
      java.lang.RuntimeException: Unknown code 98
       at org.apache.hadoop.hbase.KeyValue$Type.codeToType(KeyValue.java:274)
       at org.apache.hadoop.hbase.CellUtil.getCellKeyAsString(CellUtil.java:1307)
       at org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.getMidpoint(HFileWriterImpl.java:383)
       at org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.finishBlock(HFileWriterImpl.java:343)
       at org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.close(HFileWriterImpl.java:603)
       at org.apache.hadoop.hbase.regionserver.StoreFileWriter.close(StoreFileWriter.java:376)
       at org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.abortWriter(DefaultCompactor.java:98)
       at org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.abortWriter(DefaultCompactor.java:42)
       at org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:335)
       at org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:65)
       at org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:126)
       at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1429)
       at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2231)
       at org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:629)
       at org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:671)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)
      
      

        Attachments

        1. HBASE-22433-master-v1.patch
          16 kB
          Lijin Bin

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                binlijin Lijin Bin
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated: