[HDFS-3900] QJM: avoid validating log segments on log rolls

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: QuorumJournalManager (HDFS-3077)
• Fix Version/s: QuorumJournalManager (HDFS-3077)
• Target Version/s: QuorumJournalManager (HDFS-3077)
• Hadoop Flags: Reviewed
• Assignee: Todd Lipcon
• Reporter: Todd Lipcon
• Created: 07/Sep/12 03:24
• Updated: 10/Sep/12 18:54
• Resolved: 10/Sep/12 18:54

Description

Currently, we are paranoid and validate every log segment when it is finalized. For the a log segment that has been written entirely by one writer, with no recovery in between, this is overly paranoid (we don't do this for local journals). It also causes log rolls to be slow and take time linear in the size of the segment. Instead, we should optimize this path to simply trust that the segment is correct so long as the txids match up as expected.