[HDDS-4656] Add a container balancer tool or service for HDDS

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.2.0
• Component/s: Ozone Datanode, SCM, Tools
• Assignee: Baolong Mao
• Reporter: Baolong Mao
• Created: 08/Jan/21 00:46
• Updated: 04/Jan/24 11:40
• Resolved: 31/Mar/23 15:09

Description

When an existing Ozone cluster is nearly full, we have to add more datanodes into the Ozone cluster, but there are two issue we must face.

• When new allocate container request coming, SCM should better to choose the datanodes in low usage, if not, the performance will getting pool.
• For read request, the existing datanodes stored lots of blocks, so they are responsible for serving the read request and supply the data stream service, meanwhile, the new coming datanodes can help nothing.

If we have a balancer tool just like hdfs balancer, we can move the block or container from some high usage datanodes to low, I think this is one of necessary tools for Ozone.

container balancer design doc https://docs.google.com/document/d/15PdYaP6aLB18ptbcOK3XWlL4Y1PLfIP-ll30KjNPZ2g/edit?usp=sharing

Sub-Tasks

• Introduce ContainerBalancer in SCM with start/stop capabilities.: Resolved
• Support start/stop for container balancer via command line: Resolved
• Determine over and under utilized datanodes in Container Balancer: Resolved
• Support container move in Replication Manager: Resolved
• Select target datanodes and containers to move for Container Balancer: Resolved
• ContainerBalancer should use remaining space to calculate utilization.: Resolved
• Support container move HA: Resolved
• Balancer iterations should run on updated utilisation info from datanodes: Resolved
• ContainerBalancer#checkConditionsForBalancing pre-emptively checks iteration limits.: Resolved
• make it configurable to choose the nearest one as the target in the candidates according to networkTopology: Resolved
• Stop ContainerBalancer when SCM is stopped.: Resolved
• support Optional<T> as parameters of commandLine: Resolved
• ContainerBalancer#stop should prevent the current balancing thread from interrupting itself.: Resolved
• ContainerBalancer should get OzoneConfiguration from ContainerBalancerConfiguration.: Resolved
• support -1 for running balancer infinitely: Resolved
• make it configurable to trigger refresh datanode usage info before start a new balance iteration: Resolved
• support setting maxSizeEnteringTarget and maxSizeLeavingSource in command line: Resolved
• Incorrect calculation of iteration related metrics in ContainerBalancer: Resolved
• add a command to trigger datanode executing "du" immediately: Resolved
• Reset default values in ContainerBalancerConfiguration: Resolved
• balancer should stop when the cluster can not be balanced any more: Resolved
• ContainerBalancer incorrectly exits an iteration early without checking move results.: Resolved
• Support setting Datanode Reserved Space in MiniOzoneCluster.: Resolved
• Fix ContainerBalancerConfiguration annotation string: Resolved
• Support configuration for including/excluding datanodes for balancing: Resolved
• Improve defaults in ContainerBalancerConfiguration: Resolved
• ContainerBalancerConfig doesn't read config from ozone-site.xml: Resolved
• sourceToTargetMap in ContainerBalancer doesn't support multiple entries with same source.: Resolved
• ContainerBalancer metrics don't show updated values in JMX: Resolved
• Support Container Balancer HA: Resolved
• Add aggregate metrics to ContainerBalancerMetrics: Resolved
• ContainerBalancer shows incorrect iteration result sometimes: Resolved
• Implement ContainerBalancer as an SCMService: Resolved
• Add metric for failed container moves: Resolved
• Test "size.leaving.source.max" limit in ContainerBalancer: Resolved
• Continuous start & stop can have hanging threads in stopping: Resolved
• ContainerBalancer need have separate class handling management and balancing activity: Resolved
• Improve logging in Container Balancer: Resolved