[HDFS-3797] QJM: add segment txid as a parameter to journal() RPC

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: QuorumJournalManager (HDFS-3077)
• Fix Version/s: QuorumJournalManager (HDFS-3077)
• Component/s: ha
• Target Version/s: QuorumJournalManager (HDFS-3077)
• Hadoop Flags: Reviewed
• Assignee: Todd Lipcon
• Reporter: Todd Lipcon
• Created: 13/Aug/12 22:40
• Updated: 15/Aug/12 19:14
• Resolved: 15/Aug/12 19:14

Description

During fault testing of QJM, I saw the following issue:

1) NN sends txn 5 to JN
2) NN gets partitioned from JN while JN remains up. The next two RPCs are missed while the partition has happened:
2a) finalizeSegment(1-5)
2b) startSegment(6)
3) NN sends txn 6 to JN

This caused one of the JNs to end up with a segment 1-10 while the others had two segments; 1-5 and 6-10. This broke some invariants of the QJM protocol and prevented the recovery protocol from running properly.

This can be addressed on the client side by HDFS-3726, which would cause the NN to not send the RPC in #3. But it makes sense to also add an extra safety check here on the server side: with every journal() call, we can send the segment's txid. Then if the JN and the client get "out of sync", the JN can reject the RPCs.