[HDFS-14370] Edit log tailing fast-path should allow for backoff

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 2.10.0, 3.0.4, 3.3.0, 3.2.1, 3.1.3
• Component/s: namenode, qjm
• Target Version/s: 3.3.0
• Assignee: Erik Krogen
• Reporter: Erik Krogen
• Created: 13/Mar/19 20:49
• Updated: 02/Oct/19 17:14
• Resolved: 13/Aug/19 15:32

Description

As part of HDFS-13150, in-progress edit log tailing was changed to use an RPC-based mechanism, thus allowing the edit log tailing frequency to be turned way down, and allowing standby/observer NameNodes to be only a few milliseconds stale as compared to the Active NameNode.

When there is a high volume of transactions on the system, each RPC fetches transactions and takes some time to process them, self-rate-limiting how frequently an RPC is submitted. In a lightly loaded cluster, however, most of these RPCs return an empty set of transactions, consuming a high (de)serialization overhead for very little benefit. This was reported by jojochuang in HDFS-14276 and I have also seen it on a test cluster where the SbNN was submitting 8000 RPCs per second that returned empty.

I propose we add some sort of backoff to the tailing, so that if an empty response is received, it will wait a longer period of time before submitting a new RPC.

Issue Links

• is related to: HDFS-14276 [SBN read] Reduce tailing overhead