[HDFS-13150] [Edit Tail Fast Path] Allow SbNN to tail in-progress edits from JN via RPC

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: HDFS-12943, 3.3.0
• Component/s: ha, hdfs, journal-node, namenode
• Assignee: Erik Krogen
• Reporter: Erik Krogen
• Created: 15/Feb/18 17:32
• Updated: 04/Mar/22 16:47
• Resolved: 17/Jul/18 18:37

Description

In the interest of making coordinated/consistent reads easier to complete with low latency, it is advantageous to reduce the time between when a transaction is applied on the ANN and when it is applied on the SbNN. We propose adding a new "fast path" which can be used to tail edits when low latency is desired. We leave the existing tailing logic in place, and fall back to this path on startup, recovery, and when the fast path encounters unrecoverable errors.

Issue Links

• depends upon: HDFS-13607 [Edit Tail Fast Path Pt 1] Enhance JournalNode with an in-memory cache of recent edit transactions
• depends upon: HDFS-13608 [Edit Tail Fast Path Pt 2] Add ability for JournalNode to serve edits via RPC
• depends upon: HDFS-13609 [Edit Tail Fast Path Pt 3] NameNode-side changes to support tailing edits via RPC
• depends upon: HDFS-13610 [Edit Tail Fast Path Pt 4] Cleanup: integration test, documentation, remove unnecessary dummy sync
• is related to: HADOOP-16208 Do Not Log InterruptedException in Client
• relates to: HDFS-13789 Reduce logging frequency of QuorumJournalManager#selectInputStreams
• relates to: HDFS-16493 [SBN Read]When fast path tail enabled, standby or observer namenode may read uncommitted data
• relates to: HDFS-14500 NameNode StartupProgress continues to report edit log segments after the LOADING_EDITS phase is finished