[HDDS-7089] Intermittent failure in freon data validation after datanode restart in Kubernetes test

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 1.4.0
• Component/s: test
• Labels: kubernetes
• Target Version/s: 1.4.0
• Assignee: Attila Doroszlai
• Reporter: Attila Doroszlai
• Created: 03/Aug/22 21:49
• Updated: 17/Feb/23 16:43
• Resolved: 17/Feb/23 16:43

Description

Kubernetes check intermittently fails in ozone environment during Freon validation after datanode restart.

https://github.com/adoroszlai/ozone-build-results/blob/47b61124be7b5048334455d4a328a21d959ad334/2022/08/03/16519/kubernetes/output.log#L697-L726

pod "datanode-0" deleted
pod "datanode-1" deleted
pod "datanode-2" deleted


**** Waiting until the k8s cluster is running ****


...
2022-08-03 13:36:17 INFO  SCMSafeModeManager:248 - SCM exiting safe mode.


**** Cluster is up and running ****




**** Executing robot tests scm-0 ****


...
==============================================================================
Validate :: Test freon data validation commands                               
==============================================================================
Ozone Client Key Validator                                            | FAIL |
255 != 0

Issue Links

• is a child of: HDDS-5626 Track and Address Flaky tests
• is caused by: HDDS-5916 DNs in pipeline raft group get stuck in infinite leader election in Kubernets env
• is duplicated by: HDDS-7317 CI: kubernetes fails intermittently in freon data validation
• is fixed by: HDDS-7231 Integrate the GetKeyInfo API to key read flows