[HDFS-16872] Fix log throttling by declaring LogThrottlingHelper as static members

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.4
• Fix Version/s: 3.4.0, 3.2.5, 3.3.6
• Component/s: namanode
• Labels: pull-request-available
• Target Version/s: 3.4.0, 3.3.6
• Hadoop Flags: Reviewed
• Assignee: Chengbing Liu
• Reporter: Chengbing Liu
• Created: 19/Dec/22 12:22
• Updated: 28/Jan/24 07:32
• Resolved: 10/Jan/23 18:31

Description

In our production cluster with Observer NameNode enabled, we have plenty of logs printed by FSEditLogLoader and RedundantEditLogInputStream. The LogThrottlingHelper doesn't seem to work.

2022-10-25 09:26:50,380 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Start loading edits file ByteStringEditLog[17686250688, 17686250688], ByteStringEditLog[17686250688, 17686250688], ByteStringEditLog[17686250688, 17686250688] maxTxnsToRead = 9223372036854775807
2022-10-25 09:26:50,380 INFO org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream: Fast-forwarding stream 'ByteStringEditLog[17686250688, 17686250688], ByteStringEditLog[17686250688, 17686250688], ByteStringEditLog[17686250688, 17686250688]' to transaction ID 17686250688
2022-10-25 09:26:50,380 INFO org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream: Fast-forwarding stream 'ByteStringEditLog[17686250688, 17686250688]' to transaction ID 17686250688
2022-10-25 09:26:50,380 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Loaded 1 edits file(s) (the last named ByteStringEditLog[17686250688, 17686250688], ByteStringEditLog[17686250688, 17686250688], ByteStringEditLog[17686250688, 17686250688]) of total size 527.0, total edits 1.0, total load time 0.0 ms

2022-10-25 09:26:50,387 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Start loading edits file ByteStringEditLog[17686250689, 17686250693], ByteStringEditLog[17686250689, 17686250693], ByteStringEditLog[17686250689, 17686250693] maxTxnsToRead = 9223372036854775807
2022-10-25 09:26:50,387 INFO org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream: Fast-forwarding stream 'ByteStringEditLog[17686250689, 17686250693], ByteStringEditLog[17686250689, 17686250693], ByteStringEditLog[17686250689, 17686250693]' to transaction ID 17686250689
2022-10-25 09:26:50,387 INFO org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream: Fast-forwarding stream 'ByteStringEditLog[17686250689, 17686250693]' to transaction ID 17686250689
2022-10-25 09:26:50,387 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Loaded 1 edits file(s) (the last named ByteStringEditLog[17686250689, 17686250693], ByteStringEditLog[17686250689, 17686250693], ByteStringEditLog[17686250689, 17686250693]) of total size 890.0, total edits 5.0, total load time 1.0 ms

After some digging, I found the cause is that LogThrottlingHelper's are declared as instance variables of all the enclosing classes, including FSImage, FSEditLogLoader and RedundantEditLogInputStream. Therefore the logging frequency will not be limited across different instances. For classes with only limited number of instances, such as FSImage, this is fine. For others whose instances are created frequently, such as FSEditLogLoader and RedundantEditLogInputStream, it will result in plenty of logs.

This can be fixed by declaring LogThrottlingHelper's as static members.

Issue Links

• is caused by: HDFS-13791 Limit logging frequency of edit tail related statements
• links to: GitHub Pull Request #5246
• links to: GitHub Pull Request #5269