Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-1880 Decommissioning and maintenance mode in Ozone
  3. HDDS-2860

Cluster disk space metrics should reflect decommission and maintenance states

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.5.0
    • None
    • SCM

    Description

      Now we have decommission states, we need to adjust the cluster capacity, space used and available metrics which are exposed via JMX.

      For a node decommissioning, the space used on the node effectively needs to be transfer to other nodes via container replication before decommission can complete, but this is difficult to track from a space usage perspective. When a node completes decommission, we can assume it provides no capacity to the cluster and uses none. Therefore, for decommissioning + decommissioned nodes, the simplest calculation is to exclude the node completely in a similar way to a dead node.

      For maintenance nodes, things are even less clear. For a maintenance node, it is read only so it cannot provide capacity to the cluster, but it is expected to return to service, so excluding it completely probably does not make sense. However, perhaps the simplest solution is to do the following:

      1. For any node not IN_SERVICE, do not include its usage or space in the cluster capacity totals.
      2. Introduce some new metrics to account for the maintenance and perhaps decommission capacity, so it is not lost eg:

      # Existing metrics
      "DiskCapacity" : 62725623808,
      "DiskUsed" : 4096,
      "DiskRemaining" : 50459619328,
      
      # Suggested additional new ones, with the above only considering IN_SERVICE nodes:
      
      "MaintenanceDiskCapacity": 0
      "MaintenanceDiskUsed": 0
      "MaintenanceDiskRemaining": 0
      "DecommissionedDiskCapacity": 0
      "DecommissionedDiskUsed": 0
      "DecommissionedDiskRemaining": 0
      ...
      

      That way, the cluster totals are only what is currently "online", but we have the other metrics to track what has been removed etc. The key advantage of this, is that it is easy to understand.

      There could also be an argument that the new decommissionedDisk metrics are not needed as that capacity is technically lost from the cluster forever.

      Attachments

        Issue Links

          Activity

            People

              sodonnell Stephen O'Donnell
              sodonnell Stephen O'Donnell
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 10m
                  10m