[HDDS-9542] ozone debug chunkinfo command shows incorrect number of entries

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: Tools
• Labels: pull-request-available
• Assignee: Aryan Gupta
• Reporter: Pratyush Bhatt
• Created: 26/Oct/23 05:28
• Updated: 30/Nov/23 11:22
• Resolved: 30/Nov/23 11:21

Description

Scenario: Stop a DN from the pipeline and check re-replication.

Steps:
1. Do a key put.
2. List all the replica DNs.
3. Stop one of the DN.
4. Do "ozone debug chuninfo" command.

Configs:

"hdds.scm.replication.thread.interval": "5m",
"ozone.scm.stale.node.interval": "2m",
"ozone.scm.dead.node.interval": "4m", 

Observed behavior:
The command is throwing ERROR scm.XceiverClientGrpc: Failed to execute command GetBlock and it lists only 1 Keylocation(sometimes 0).

[root@ozn-decom75-1 ~]# ozone debug chunkinfo /testv1/testb1/passwd
23/10/26 03:30:34 WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-xceiverclientmetrics.properties,hadoop-metrics2.properties
23/10/26 03:30:34 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
23/10/26 03:30:34 INFO impl.MetricsSystemImpl: XceiverClientMetrics metrics system started
23/10/26 03:30:35 ERROR scm.XceiverClientGrpc: Failed to execute command GetBlock. Exception Class: java.util.concurrent.ExecutionException, Exception Message: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
{
  "KeyLocations": [
    [
      {
        "Datanode-HostName": "ozn-decom75-9.ozn-decom75.xyz",
        "Datanode-IP": "172.27.139.192",
        "Container-ID": 12002,
        "Block-ID": 111677748019344024,
        "Locations": {
          "files": [
            "/hadoop-ozone/datanode/data184295/hdds/CID-d16eaa18-863d-4914-8769-bc3763e7e83c/current/containerDir23/12002/chunks/111677748019344024.block"
          ],
          "pipelineID": "9cfb4d55-a00d-4231-aeb6-cb63b8e26b3c"
        }
      }
    ]
  ]
} 

This happens till the dead node interval is met. After that it shows 2 Keylocations(The DNs that are up from that pipeline)

[root@ozn-decom75-1 ~]# ozone debug chunkinfo /testv1/testb1/passwd
23/10/26 03:34:22 WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-xceiverclientmetrics.properties,hadoop-metrics2.properties
23/10/26 03:34:22 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
23/10/26 03:34:22 INFO impl.MetricsSystemImpl: XceiverClientMetrics metrics system started
{
  "KeyLocations": [
    [
      {
        "Datanode-HostName": "ozn-decom75-9.ozn-decom75.xyz",
        "Datanode-IP": "172.27.139.192",
        "Container-ID": 12002,
        "Block-ID": 111677748019344024,
        "Locations": {
          "files": [
            "/hadoop-ozone/datanode/data184295/hdds/CID-d16eaa18-863d-4914-8769-bc3763e7e83c/current/containerDir23/12002/chunks/111677748019344024.block"
          ],
          "pipelineID": "7b36169b-6f94-4e86-a8f7-4c9091cb3f7a"
        }
      },
      {
        "Datanode-HostName": "ozn-decom75-3.ozn-decom75.xyz",
        "Datanode-IP": "172.27.198.65",
        "Container-ID": 12002,
        "Block-ID": 111677748019344024,
        "Locations": {
          "files": [
            "/hadoop-ozone/datanode/data184295/hdds/CID-d16eaa18-863d-4914-8769-bc3763e7e83c/current/containerDir23/12002/chunks/111677748019344024.block"
          ],
          "pipelineID": "7b36169b-6f94-4e86-a8f7-4c9091cb3f7a"
        }
      }
    ]
  ]
} 

Issue Links

• links to: GitHub Pull Request #5703