[HDFS-4025] QJM: Sychronize past log segments to JNs that missed them

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: QuorumJournalManager (HDFS-3077)
• Fix Version/s: QuorumJournalManager (HDFS-3077), 3.0.0-alpha4
• Component/s: ha
• Hadoop Flags: Reviewed
• Assignee: Hanisha Koneru
• Reporter: Todd Lipcon
• Created: 10/Oct/12 17:48
• Updated: 30/Oct/19 07:33
• Resolved: 23/Feb/17 00:37

Description

Currently, if a JournalManager crashes and misses some segment of logs, and then comes back, it will be re-added as a valid part of the quorum on the next log roll. However, it will not have a complete history of log segments (i.e any individual JN may have gaps in its transaction history). This mirrors the behavior of the NameNode when there are multiple local directories specified.

However, it would be better if a background thread noticed these gaps and "filled them in" by grabbing the segments from other JournalNodes. This increases the resilience of the system when JournalNodes get reformatted or otherwise lose their local disk.

Issue Links

• blocks: HDFS-10659 Namenode crashes after Journalnode re-installation in an HA cluster due to missing paxos directory
• is blocked by: HDFS-11273 Move TransferFsImage#doGetUrl function to a Util class
• is related to: HDFS-12358 Handle IOException when transferring edit log to Journal current dir through JN sync
• is related to: HDFS-12356 Unit test for JournalNode sync during Rolling Upgrade
• is related to: HDFS-14942 Change Log Level to debug in JournalNodeSyncer#syncWithJournalAtIndex
• relates to: HDFS-11448 JN log segment syncing should support HA upgrade
• relates to: HDFS-11866 JournalNode Sync should be off by default in hdfs-default.xml
• relates to: HDFS-14140 JournalNodeSyncer authentication is failing in secure cluster
• relates to: HDFS-12376 Enable JournalNode Sync by default