[HDDS-7317] CI: kubernetes fails intermittently in freon data validation

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Assignee: Unassigned
• Reporter: Siyao Meng
• Created: 12/Oct/22 18:24
• Updated: 05/Oct/23 12:16
• Resolved: 12/Oct/22 18:42

Description

The following is from duongn 's PR CI. I recall seeing this somewhere else recently (and frequently) in other PRs as well.

https://github.com/apache/ozone/actions/runs/3230990324/jobs/5290112904#step:5:835

Validate :: Test freon data validation commands                               
==============================================================================
Ozone Client Key Validator                                            | FAIL |
255 != 0
------------------------------------------------------------------------------
Validate :: Test freon data validation commands                       | FAIL |
1 test, 0 passed, 1 failed

In log.html :

2022-10-12 00:16:59 INFO  ProgressBar:163 - Progress: 0.00 % (0 out of 1)
2022-10-12 00:17:00 INFO  ProgressBar:163 - Progress: 0.00 % (0 out of 1)
2022-10-12 00:17:01 ERROR XceiverClientGrpc:448 - Failed to execute command GetBlock on the pipeline Pipeline[ Id: 6c3d5f08-eb29-45ad-a4e8-376820e6cd4d, Nodes: a33df79e-5c22-4b80-8c3e-7534be98d40f{ip: 10.42.0.38, host: datanode-1.datanode.default.svc.cluster.local, ports: [REPLICATION=9886, RATIS=9858, RATIS_ADMIN=9857, RATIS_SERVER=9856, STANDALONE=9859], networkLocation: /default-rack, certSerialId: null, persistedOpState: IN_SERVICE, persistedOpStateExpiryEpochSec: 0}5c1169c0-c126-4179-925a-5c4eaed33ef1{ip: 10.42.0.40, host: datanode-2.datanode.default.svc.cluster.local, ports: [REPLICATION=9886, RATIS=9858, RATIS_ADMIN=9857, RATIS_SERVER=9856, STANDALONE=9859], networkLocation: /default-rack, certSerialId: null, persistedOpState: IN_SERVICE, persistedOpStateExpiryEpochSec: 0}dc9aa0e3-e724-456b-b0e9-4eb442904813{ip: 10.42.0.36, host: datanode-0.datanode.default.svc.cluster.local, ports: [REPLICATION=9886, RATIS=9858, RATIS_ADMIN=9857, RATIS_SERVER=9856, STANDALONE=9859], networkLocation: /default-rack, certSerialId: null, persistedOpState: IN_SERVICE, persistedOpStateExpiryEpochSec: 0}, ReplicationConfig: STANDALONE/THREE, State:OPEN, leaderId:5c1169c0-c126-4179-925a-5c4eaed33ef1, CreationTimestamp2022-10-12T00:14:30.358Z[UTC]].
java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception

cc erose adoroszlai

Issue Links

• duplicates: HDDS-7089 Intermittent failure in freon data validation after datanode restart in Kubernetes test