Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-9542

ozone debug chunkinfo command shows incorrect number of entries

    XMLWordPrintableJSON

Details

    Description

      Scenario: Stop a DN from the pipeline and check re-replication.

      Steps:
      1. Do a key put.
      2. List all the replica DNs.
      3. Stop one of the DN.
      4. Do "ozone debug chuninfo" command.

      Configs:

      "hdds.scm.replication.thread.interval": "5m",
      "ozone.scm.stale.node.interval": "2m",
      "ozone.scm.dead.node.interval": "4m", 

      Observed behavior:
      The command is throwing ERROR scm.XceiverClientGrpc: Failed to execute command GetBlock and it lists only 1 Keylocation(sometimes 0).

      [root@ozn-decom75-1 ~]# ozone debug chunkinfo /testv1/testb1/passwd
      23/10/26 03:30:34 WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-xceiverclientmetrics.properties,hadoop-metrics2.properties
      23/10/26 03:30:34 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
      23/10/26 03:30:34 INFO impl.MetricsSystemImpl: XceiverClientMetrics metrics system started
      23/10/26 03:30:35 ERROR scm.XceiverClientGrpc: Failed to execute command GetBlock. Exception Class: java.util.concurrent.ExecutionException, Exception Message: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
      {
        "KeyLocations": [
          [
            {
              "Datanode-HostName": "ozn-decom75-9.ozn-decom75.xyz",
              "Datanode-IP": "172.27.139.192",
              "Container-ID": 12002,
              "Block-ID": 111677748019344024,
              "Locations": {
                "files": [
                  "/hadoop-ozone/datanode/data184295/hdds/CID-d16eaa18-863d-4914-8769-bc3763e7e83c/current/containerDir23/12002/chunks/111677748019344024.block"
                ],
                "pipelineID": "9cfb4d55-a00d-4231-aeb6-cb63b8e26b3c"
              }
            }
          ]
        ]
      } 

      This happens till the dead node interval is met. After that it shows 2 Keylocations(The DNs that are up from that pipeline)

      [root@ozn-decom75-1 ~]# ozone debug chunkinfo /testv1/testb1/passwd
      23/10/26 03:34:22 WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-xceiverclientmetrics.properties,hadoop-metrics2.properties
      23/10/26 03:34:22 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
      23/10/26 03:34:22 INFO impl.MetricsSystemImpl: XceiverClientMetrics metrics system started
      {
        "KeyLocations": [
          [
            {
              "Datanode-HostName": "ozn-decom75-9.ozn-decom75.xyz",
              "Datanode-IP": "172.27.139.192",
              "Container-ID": 12002,
              "Block-ID": 111677748019344024,
              "Locations": {
                "files": [
                  "/hadoop-ozone/datanode/data184295/hdds/CID-d16eaa18-863d-4914-8769-bc3763e7e83c/current/containerDir23/12002/chunks/111677748019344024.block"
                ],
                "pipelineID": "7b36169b-6f94-4e86-a8f7-4c9091cb3f7a"
              }
            },
            {
              "Datanode-HostName": "ozn-decom75-3.ozn-decom75.xyz",
              "Datanode-IP": "172.27.198.65",
              "Container-ID": 12002,
              "Block-ID": 111677748019344024,
              "Locations": {
                "files": [
                  "/hadoop-ozone/datanode/data184295/hdds/CID-d16eaa18-863d-4914-8769-bc3763e7e83c/current/containerDir23/12002/chunks/111677748019344024.block"
                ],
                "pipelineID": "7b36169b-6f94-4e86-a8f7-4c9091cb3f7a"
              }
            }
          ]
        ]
      } 

      Attachments

        Issue Links

          Activity

            People

              aryangupta1998 Aryan Gupta
              pratyush.bhatt Pratyush Bhatt
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: