[HDFS-3077] Quorum-based protocol for reading and writing edit logs

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: QuorumJournalManager (HDFS-3077), 2.0.3-alpha
• Component/s: ha, namenode
• Target Version/s: QuorumJournalManager (HDFS-3077)
• Hadoop Flags: Reviewed
• Assignee: Todd Lipcon
• Reporter: Todd Lipcon
• Created: 12/Mar/12 18:44
• Updated: 16/Jun/16 22:54
• Resolved: 05/Dec/12 19:22

Description

Currently, one of the weak points of the HA design is that it relies on shared storage such as an NFS filer for the shared edit log. One alternative that has been proposed is to depend on BookKeeper, a ZooKeeper subproject which provides a highly available replicated edit log on commodity hardware. This JIRA is to implement another alternative, based on a quorum commit protocol, integrated more tightly in HDFS and with the requirements driven only by HDFS's needs rather than more generic use cases. More details to follow.

Issue Links

• is depended upon by: HDFS-3278 Umbrella Jira for HDFS-HA Phase 2
• is related to: HDFS-3752 BOOTSTRAPSTANDBY for new Standby node will not work just after saveNameSpace at ANN in case of BKJM
• is related to: HDFS-3765 Namenode INITIALIZESHAREDEDITS should be able to initialize all shared storages
• relates to: HDFS-4045 SecondaryNameNode cannot read from QuorumJournal URI
• relates to: HDFS-3092 Enable journal protocol based editlog streaming for standby namenode

Sub-Tasks

• Upgrade guava to 11.0.2: Closed
• Add infrastructure for waiting for a quorum of ListenableFutures to respond: Resolved
• Add preliminary QJournalProtocol interface, translators: Resolved
• Simple refactors in existing NN code to assist QuorumJournalManager extension: Closed
• Allow EditLogFileInputStream to read from a remote URL: Closed
• Supply NamespaceInfo when instantiating JournalManagers: Closed
• Active NN should exit when it cannot write to quorum number of Journal Daemons: Resolved
• Add class to manage JournalList: Resolved
• QJM: support purgeEditLogs() call to remotely purge logs: Resolved
• QJM: JNStorage should read its storage info even before a writer becomes active: Resolved
• QJM: Fix getEditLogManifest to fetch httpPort if necessary: Resolved
• Genericize format() to non-file JournalManagers: Closed
• Fix QJM startup when individual JNs have gaps: Resolved
• QJM: if a logger misses an RPC, don't retry that logger until next segment: Resolved
• QJM: exhaustive failure injection test for skipped RPCs: Resolved
• QJM: improve formatting behavior for JNs: Open
• JournalManager#format() should be able to throw IOException: Closed
• Implement genericized format() in QJM: Resolved
• QJM: validate journal dir at startup: Resolved
• QJM: add segment txid as a parameter to journal() RPC: Resolved
• Avoid throwing NPE when finalizeSegment() is called on invalid segment: Resolved
• QJM: handle empty log segments during recovery: Resolved
• QJM: improvements to QJM fault testing: Resolved
• QJM: hadoop-daemon.sh should be updated to accept "journalnode": Resolved
• Fixes for edge cases in QJM recovery protocol: Resolved
• QJM: implement md5sum verification: Open
• QJM: don't require a fencer to be configured if shared storage has built-in single-writer semantics: Patch Available
• QJM: track last "committed" txid: Resolved
• QJM: Support rolling restart of JNs: Open
• QJM: expose non-file journal manager details in web UI: Resolved
• QJM: add metrics to JournalNode: Resolved
• QJM: Provide defaults for dfs.journalnode.*address: Resolved
• QJM: Journal format() should reset cached values: Resolved
• QJM: optimize log sync when JN is lagging behind: Resolved
• QJM: SBN fails if selectInputStreams throws RTE: Resolved
• QJM: Make QJM work with security enabled: Resolved
• QJM: testRecoverAfterDoubleFailures can be flaky due to IPC client caching: Resolved
• QJM: enable TCP_NODELAY for IPC: Resolved
• QJM: Writer-side metrics: Resolved
• QJM: avoid validating log segments on log rolls: Resolved
• QJM: send 'heartbeat' messages to JNs even when they are out-of-sync: Resolved
• QJM: journalnode does not die/log ERROR when keytab is not found in secure mode: Resolved
• QJM: quorum timeout on failover with large log segment: Resolved
• QJM: acceptRecovery should abort current segment: Resolved
• QJM: Failover fails with auth error in secure cluster: Resolved
• JournalNodes log JournalNotFormattedException backtrace error before being formatted: Resolved
• QJM: Add user documentation for QJM: Resolved
• QJM: Add JournalNode to the start / stop scripts: Closed
• QJM: remove currently unused "md5sum" field.: Resolved
• QJM: misc TODO cleanup, improved log messages, etc: Resolved
• QJM: Make acceptRecovery() atomic: Resolved
• QJM: purge temporary files when no longer within retention period: Resolved
• TestJournalNode#testJournal fails because of test case execution order: Resolved
• Unclosed FileInputStream in GetJournalEditServlet: Resolved
• QJM: Sychronize past log segments to JNs that missed them: Resolved
• QJM: Merge newEpoch and prepareRecovery: Open