Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-7759 Improve Ozone Replication Manager
  3. HDDS-8505

ReplicationManager: Add configurable global replication limit

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Implemented
    • None
    • 1.4.0
    • SCM

    Description

      We should make it possible to configure a global replication limit, limiting the number of inflight containers pending creation. A larger cluster would be capable of having more inflight replication than a smaller cluster, so the limit should be a function of the number of datanodes on the cluster, and the limit of the number of commands which can be queued per datanode and some weighting factor.

      For example, if each datanode can queue 20 replication commands, and there are 100 nodes in the cluster, then the natural limit is 20 * 100. However, that assumes that commands are queued evenly across all datanodes, which is unlikely. With a global limit we would prefer that all datanodes are not fully loaded with replication commands simultaneously, so we may want to impose a limit of half that number, with a factor of 0.5, eg 20 * 100 * 0.5 = 1k pending replications.

      At one extreme this would result in all datanodes in the cluster having half their maximum tasks queued, but in practice, some DNs are likely to be at their limit while others have zero or less than half queued.

      If the limits were perfectly defined, such that in a single heartbeat a datanode can complete all its queued work just at the end of the heartbeat interval, then reducing the number of queued commands by half would make the datanode busy for only half its heartbeat interval. As the datanodes will all heartbeat at different times, all the busy and non-work periods across all the datanodes would combine in a load profile that would show some datanodes are always idle, reducing the overall load on the cluster.

      Attachments

        Issue Links

          Activity

            People

              sodonnell Stephen O'Donnell
              sodonnell Stephen O'Donnell
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: