[HDDS-4539] Container Health Task should not run until Recon has reached steady state.

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: Ozone Recon
• Labels: pull-request-available
• Assignee: Devesh Kumar Singh
• Reporter: Aravindan Vijayan
• Created: 02/Dec/20 19:32
• Updated: 09/Jan/23 19:25
• Resolved: 09/Jan/23 19:24

Description

On a cluster with millions of containers or hundreds of Datanodes, it will take some time for Recon to reach a steady state (all active DNs and Containers reported). If the container health task is run before this, it can incorrectly flag most of the containers as missing. This was seen in a cluster where Recon reaching steady state is slow due to HDDS-4403, and it also leads to the UI problem mentioned in HDDS-4402.

We need to make sure the container health task is not run before cluster has reached steady state. This could be a fixed wait time (~10mins) or by checking Recon's SCM state.

Issue Links

• links to: GitHub Pull Request #4049