Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-5885

Invalid HFile block magic on Local file System

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.94.0, 0.95.2
    • Fix Version/s: 0.94.0, 0.95.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      ERROR: java.lang.RuntimeException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=7, exceptions:
      Thu Apr 26 11:19:18 PDT 2012, org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for reader reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268, compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], firstKey=0000000001/info:data/1335463981520/Put, lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, avgValueLen=1000, entries=1215085, length=1264354417, cur=0000000248/info:data/1335463994457/Put/vlen=1000/ts=0]
      at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135)
      at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95)
      at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368)
      at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
      at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323)
      at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279)
      at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393)
      at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
      at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
      Caused by: java.io.IOException: Invalid HFile block magic: \xEC\xD5\x9D\xB4\xC2bfo
      at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153)
      at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164)
      at org.apache.hadoop.hbase.io.hfile.HFileBlock.<init>(HFileBlock.java:254)
      at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779)
      at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637)
      at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327)
      at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555)
      at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651)
      at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
      ... 12 more

      Thu Apr 26 11:19:19 PDT 2012, org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: java.io.IOException: java.lang.IllegalArgumentException
      at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420)
      at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
      at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
      Caused by: java.lang.IllegalArgumentException
      at java.nio.Buffer.position(Buffer.java:216)
      at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:630)
      at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
      at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95)
      at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:406)
      at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
      at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323)
      at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279)
      at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393)
      ... 5 more

      On latest 0.94 branch I spun up a new standalone hbase. Then I started a performance evaluation run hbase/bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred randomWrite 10
      after that completed I tried a scan of TestTable. The scan got the above error.

        Attachments

        1. HBASE-5885-94-0.patch
          1 kB
          Elliott Clark
        2. HBASE-5885-trunk-0.patch
          1 kB
          Elliott Clark
        3. HBASE-5885-94-1.patch
          2 kB
          Elliott Clark
        4. HBASE-5885-trunk-1.patch
          0.7 kB
          Elliott Clark
        5. 5885-trunk-v2.txt
          2 kB
          Ted Yu

          Issue Links

            Activity

              People

              • Assignee:
                eclark Elliott Clark
                Reporter:
                eclark Elliott Clark
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: