Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-1880 Decommissioning and maintenance mode in Ozone
  3. HDDS-2592

Add Datanode command to allow the datanode to persist its admin state




      When the operational state of a datanode changes, an async command should be triggered to persist the new state on the datanodes. For maintenance mode, the datanode should also store the maintenance end time. The datanode will then report the new state (and optional maintenance end time) back via its heartbeat.

      The purpose of the DN persisting this information and heartbeating it back to SCM is to allow the operation state to be recovered after a SCM reboot, as SCM does not persist any of this information. It also allows "Recon" to learn the datanode states.

      If SCM is restarted, then it will forget all knowledge of the datanodes. When they register, their operational state will be reported and SCM can set it correctly.

      Outside of registration (ie during normal heartbeats), the SCM state is the source of truth for the operational state and if the DN heartbeat reports a state that is not the same as SCM, SCM should issue another command to the datanode to set its state to the SCM value. There is a chance the state miss match is due to an unprocessed command triggered by the SCM state change, but the worst case is an extra command sent to the datanode. This is a very lightweight command, so that is not an issue.

      One open question is whether to persist intermediate states on the DN. Ie for decommissioning, the DN will first persist "Decommissioning" and then transition to "Decommissioned" when SCM is satisfied all containers are replicated. It would be possible to persist both these states in turn on the datanode quite easily in turn. Or, we set the end state (Decommissioned) on the datanode and allow SCM to get the node to that state. For the latter, if SCM is restarted, then the DN will report "Decommissioned" on registration, but SCM will set its internal state to Decommissioning and then ensure all containers are replicated before transitioning the node to Decommissioned. This seems like a safer approach, but there are advantages of tracking the intermediate states on the DNs too.


        Issue Links



              sodonnell Stephen O'Donnell
              sodonnell Stephen O'Donnell
              0 Vote for this issue
              3 Start watching this issue



                Time Tracking

                  Original Estimate - Not Specified
                  Not Specified
                  Remaining Estimate - 0h
                  Time Spent - 20m