[HDFS-13791] Limit logging frequency of edit tail related statements

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: HDFS-12943, 3.3.0
• Component/s: hdfs, qjm
• Target Version/s: HDFS-12943
• Assignee: Erik Krogen
• Reporter: Erik Krogen
• Created: 03/Aug/18 18:28
• Updated: 19/Dec/22 12:23
• Resolved: 27/Sep/18 17:13

Description

There are a number of log statements that occur every time new edits are tailed by a Standby NameNode. When edits are tailing only on the order of every tens of seconds, this is fine. With the work in HDFS-13150, however, edits may be tailed every few milliseconds, which can flood the logs with tailing-related statements. We should throttle it to limit it to printing at most, say, once per 5 seconds.

We can implement logic similar to that used in HDFS-10713. This may be slightly more tricky since the log statements are distributed across a few classes.

Issue Links

• causes: HDFS-16872 Fix log throttling by declaring LogThrottlingHelper as static members
• depends upon: HADOOP-15726 Create utility to limit frequency of log statements