Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
Description
Scenario: Stop a DN from the pipeline and check re-replication.
Steps:
1. Do a key put.
2. List all the replica DNs.
3. Stop one of the DN.
4. Do "ozone debug chuninfo" command.
Configs:
"hdds.scm.replication.thread.interval": "5m", "ozone.scm.stale.node.interval": "2m", "ozone.scm.dead.node.interval": "4m",
Observed behavior:
The command is throwing ERROR scm.XceiverClientGrpc: Failed to execute command GetBlock and it lists only 1 Keylocation(sometimes 0).
[root@ozn-decom75-1 ~]# ozone debug chunkinfo /testv1/testb1/passwd 23/10/26 03:30:34 WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-xceiverclientmetrics.properties,hadoop-metrics2.properties 23/10/26 03:30:34 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s). 23/10/26 03:30:34 INFO impl.MetricsSystemImpl: XceiverClientMetrics metrics system started 23/10/26 03:30:35 ERROR scm.XceiverClientGrpc: Failed to execute command GetBlock. Exception Class: java.util.concurrent.ExecutionException, Exception Message: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception { "KeyLocations": [ [ { "Datanode-HostName": "ozn-decom75-9.ozn-decom75.xyz", "Datanode-IP": "172.27.139.192", "Container-ID": 12002, "Block-ID": 111677748019344024, "Locations": { "files": [ "/hadoop-ozone/datanode/data184295/hdds/CID-d16eaa18-863d-4914-8769-bc3763e7e83c/current/containerDir23/12002/chunks/111677748019344024.block" ], "pipelineID": "9cfb4d55-a00d-4231-aeb6-cb63b8e26b3c" } } ] ] }
This happens till the dead node interval is met. After that it shows 2 Keylocations(The DNs that are up from that pipeline)
[root@ozn-decom75-1 ~]# ozone debug chunkinfo /testv1/testb1/passwd 23/10/26 03:34:22 WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-xceiverclientmetrics.properties,hadoop-metrics2.properties 23/10/26 03:34:22 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s). 23/10/26 03:34:22 INFO impl.MetricsSystemImpl: XceiverClientMetrics metrics system started { "KeyLocations": [ [ { "Datanode-HostName": "ozn-decom75-9.ozn-decom75.xyz", "Datanode-IP": "172.27.139.192", "Container-ID": 12002, "Block-ID": 111677748019344024, "Locations": { "files": [ "/hadoop-ozone/datanode/data184295/hdds/CID-d16eaa18-863d-4914-8769-bc3763e7e83c/current/containerDir23/12002/chunks/111677748019344024.block" ], "pipelineID": "7b36169b-6f94-4e86-a8f7-4c9091cb3f7a" } }, { "Datanode-HostName": "ozn-decom75-3.ozn-decom75.xyz", "Datanode-IP": "172.27.198.65", "Container-ID": 12002, "Block-ID": 111677748019344024, "Locations": { "files": [ "/hadoop-ozone/datanode/data184295/hdds/CID-d16eaa18-863d-4914-8769-bc3763e7e83c/current/containerDir23/12002/chunks/111677748019344024.block" ], "pipelineID": "7b36169b-6f94-4e86-a8f7-4c9091cb3f7a" } } ] ] }
Attachments
Issue Links
- links to