[HDFS-16502] Reconfigure Block Invalidate limit

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0, 3.3.5
• Fix Version/s: 3.4.0, 3.3.5
• Component/s: block placement
• Labels: pull-request-available
• Target Version/s: 3.4.0, 3.3.5
• Hadoop Flags: Reviewed
• Assignee: Viraj Jasani
• Reporter: Viraj Jasani
• Created: 12/Mar/22 09:21
• Updated: 12/Feb/24 07:40
• Resolved: 16/Mar/22 02:13

Description

Based on the cluster load, it would be helpful to consider tuning block invalidate limit (dfs.block.invalidate.limit). The only way we can do this without restarting Namenode as of today is by reconfiguring heartbeat interval 

Math.max(heartbeatInt*20, blockInvalidateLimit)

, this logic is not straightforward and operators are usually not aware of it (lack of documentation), also updating heartbeat interval is not desired in all the cases.

We should provide the ability to alter block invalidation limit without affecting heartbeat interval on the live cluster to adjust some load at Datanode level.

We should also take this opportunity to keep (heartbeatInterval * 20) computation logic in a common method.

Issue Links

• links to: GitHub Pull Request #4064