[HDFS-3077] Quorum-based protocol for reading and writing edit logs

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: QuorumJournalManager (HDFS-3077), 2.0.3-alpha
• Component/s: ha, namenode
• Target Version/s: QuorumJournalManager (HDFS-3077)
• Hadoop Flags: Reviewed
• Assignee: Todd Lipcon
• Reporter: Todd Lipcon
• Created: 12/Mar/12 18:44
• Updated: 16/Jun/16 22:54
• Resolved: 05/Dec/12 19:22

Description

Currently, one of the weak points of the HA design is that it relies on shared storage such as an NFS filer for the shared edit log. One alternative that has been proposed is to depend on BookKeeper, a ZooKeeper subproject which provides a highly available replicated edit log on commodity hardware. This JIRA is to implement another alternative, based on a quorum commit protocol, integrated more tightly in HDFS and with the requirements driven only by HDFS's needs rather than more generic use cases. More details to follow.

Issue Links

• is depended upon by: HDFS-3278 Umbrella Jira for HDFS-HA Phase 2
• is related to: HDFS-3752 BOOTSTRAPSTANDBY for new Standby node will not work just after saveNameSpace at ANN in case of BKJM
• is related to: HDFS-3765 Namenode INITIALIZESHAREDEDITS should be able to initialize all shared storages
• relates to: HDFS-4045 SecondaryNameNode cannot read from QuorumJournal URI
• relates to: HDFS-3092 Enable journal protocol based editlog streaming for standby namenode

Sub-Tasks

• QJM: improve formatting behavior for JNs: Open
• QJM: implement md5sum verification: Open
• QJM: don't require a fencer to be configured if shared storage has built-in single-writer semantics: Patch Available
• QJM: Support rolling restart of JNs: Open
• QJM: Merge newEpoch and prepareRecovery: Open