[HBASE-6586] Quarantine Corrupted HFiles with hbck

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.92.1
• Fix Version/s: 0.94.2
• Hadoop Flags: Reviewed
• Assignee: Jonathan Hsieh
• Reporter: Jonathan Hsieh
• Created: 14/Aug/12 22:03
• Updated: 07/Apr/13 05:14
• Resolved: 28/Aug/12 14:25

Description

We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.

Here's an example of the exception we've encountered:

2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d. 
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2) 
at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306) 
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371) 
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387) 
at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026) 
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485) 
at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566) 
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286) 
at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223) 
at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534) 
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454) 
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282) 
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230) 
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169) 
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) 
at java.lang.Thread.run(Thread.java:619) 
Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2) 
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515) 
at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303) 
... 17 more

Specifically – the FixedFileTrailer are incorrect, and seemingly missing.

Issue Links

• is related to: HDFS-3731 2.0 release upgrade must handle blocks being written from 1.0
• relates to: HBASE-5885 Invalid HFile block magic on Local file System
• relates to: HBASE-6600 Quarantine state for HRegions with corrupted data files