Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-6586

Quarantine Corrupted HFiles with hbck

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.92.1
    • Fix Version/s: 0.94.2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.

      Here's an example of the exception we've encountered:

      2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
      0d. 
      java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2) 
      at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306) 
      at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371) 
      at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387) 
      at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026) 
      at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485) 
      at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566) 
      at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286) 
      at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223) 
      at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534) 
      at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454) 
      at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282) 
      at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230) 
      at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
      at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
      at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169) 
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) 
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) 
      at java.lang.Thread.run(Thread.java:619) 
      Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2) 
      at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515) 
      at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303) 
      ... 17 more
      

      Specifically – the FixedFileTrailer are incorrect, and seemingly missing.

        Attachments

        1. hbase-6586.patch
          49 kB
          Jonathan Hsieh
        2. 0001-hbase-6568-hbck-quarantine-v6.patch
          57 kB
          Jonathan Hsieh
        3. hbase-6586-92-v3.patch
          54 kB
          Jonathan Hsieh
        4. hbase-6586-94-v3.patch
          56 kB
          Jonathan Hsieh
        5. hbase-6586-trunk-v3.patch
          56 kB
          Jonathan Hsieh
        6. hbase-6586-trunk-v4.patch
          57 kB
          Jonathan Hsieh
        7. hbase-6586-trunk-v5.patch
          58 kB
          Jonathan Hsieh
        8. hbase-6586-trunk-v6.patch
          58 kB
          Jonathan Hsieh
        9. hbase-6586-trunk-v7.patch
          59 kB
          Jonathan Hsieh
        10. hbase-6586-92-v8.patch
          54 kB
          Jonathan Hsieh
        11. hbase-6586-92-v8a.patch
          55 kB
          Jonathan Hsieh
        12. hbase-6586-92-v8b.patch
          55 kB
          Jonathan Hsieh
        13. hbase-6586-trunk-v8.patch
          59 kB
          Jonathan Hsieh
        14. hbase-6586-92-v9.patch
          55 kB
          Jonathan Hsieh
        15. hbase-6586-94-v9.patch
          57 kB
          Jonathan Hsieh
        16. hbase-6586-trunk-v9.patch
          58 kB
          Jonathan Hsieh

          Issue Links

            Activity

              People

              • Assignee:
                jmhsieh Jonathan Hsieh
                Reporter:
                jmhsieh Jonathan Hsieh
              • Votes:
                0 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: