[HDFS-3004] Implement Recovery Mode

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0-alpha
• Component/s: tools
• Hadoop Flags: Reviewed
• Release Note: This is a new feature. It is documented in hdfs_user_guide.xml.
• Assignee: Colin McCabe
• Reporter: Colin McCabe
• Created: 23/Feb/12 19:23
• Updated: 28/Sep/15 20:58
• Resolved: 09/Apr/12 19:47

Description

When the NameNode metadata is corrupt for some reason, we want to be able to fix it. Obviously, we would prefer never to get in this case. In a perfect world, we never would. However, bad data on disk can happen from time to time, because of hardware errors or misconfigurations. In the past we have had to correct it manually, which is time-consuming and which can result in downtime.

Recovery mode is initialized by the system administrator. When the NameNode starts up in Recovery Mode, it will try to load the FSImage file, apply all the edits from the edits log, and then write out a new image. Then it will shut down.

Unlike in the normal startup process, the recovery mode startup process will be interactive. When the NameNode finds something that is inconsistent, it will prompt the operator as to what it should do. The operator can also choose to take the first option for all prompts by starting up with the '-f' flag, or typing 'a' at one of the prompts.

I have reused as much code as possible from the NameNode in this tool. Hopefully, the effort that was spent developing this will also make the NameNode editLog and image processing even more robust than it already is.

Issue Links

• is related to: HDFS-2079 1073: create an escape hatch to ignore startup consistency problems
• is related to: HDFS-3055 Implement recovery mode for branch-1
• is related to: HDFS-3134 Harden edit log loader against malformed or malicious input
• relates to: HDFS-3049 During the normal loading NN startup process, fall back on a different EditLog if we see one that is corrupt