[HADOOP-15726] Create utility to limit frequency of log statements

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.10.0, 3.2.0, 3.0.4, 3.1.2
• Component/s: common, util
• Target Version/s: 3.0.4
• Assignee: Erik Krogen
• Reporter: Erik Krogen
• Created: 06/Sep/18 18:43
• Updated: 13/Dec/22 07:29
• Resolved: 19/Sep/18 20:31

Description

There is a common pattern of logging a behavior that is normally extraneous. Under some circumstances, such a behavior becomes common, flooding the logs and making it difficult to see what else is going on in the system. Under such situations it is beneficial to limit how frequently the extraneous behavior is logged, while capturing some summary information about the suppressed log statements.

This is currently implemented in FSNamesystemLock (in HDFS-10713). We have additional use cases for this in HDFS-13791, so this is a good time to create a common utility for different sites to share this logic.

Issue Links

• is depended upon by: HDFS-13791 Limit logging frequency of edit tail related statements
• relates to: HADOOP-18567 LogThrottlingHelper: the dependent recorder is not triggered correctly