[HBASE-1342] Add to filesystem info needed to rebuild .META.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.20.0
• Hadoop Flags: Incompatible change, Reviewed
• Assignee: Unassigned
• Reporter: Michael Stack
• Created: 23/Apr/09 22:52
• Updated: 23/Mar/10 14:00
• Resolved: 01/May/09 22:34

Description

From Cosmin up on list:

Hi,

Is there any way to rebuild the .META. Table with information in HDFS?

Long story short, the namenode ran out of disk and crashed and we couldn't
have it load the edits.new file at startup (the edits.new file seemed
corrupted and it failed to read some Long value) so we removed the edits.new
file. Apparently the edits file had 10 - 15 entries. Some edit data on
-ROOT-, some edit data on .META., some data on a table compaction
(comapction.dir) and some data on some hbase logs. Only last entry seem to
be bogus, having almost a megabyte of trailing zeroes.

After renaming edits.new HDFS started ok and HBase data is present. However
when we started Hbase one table was missing entirely and one table was
empty. The HBase master report shows Regions On FS: 94, but the report on
region servers shows only 10 regions on all the region servers.

Is there any way to cleanup hbase and rebuild the .META. table from HDFS?
This would be ok even if it implies throwing up some data.

If we would clean the edits.new file and restart HDFS or if we would
manually apply the edits would that still make consistent changes to HBase
data?

Thanks,
Cosmin

Lets log the info needed to rebuild your .META. or add things like HRegionInfo made creating a region as a file under each region directory. That don't seem hard and the info could be used rebuilding. What else would we need?

Marking as blocker/critical for 0.20.0.

Issue Links

• blocks: HBASE-7 [hbase] Provide a HBase checker and repair tool similar to fsck
• is part of: HBASE-7 [hbase] Provide a HBase checker and repair tool similar to fsck
• relates to: HBASE-2366 We should store table metadata in a single location, for HDFS table recovery, not on every region