[HDDS-3698] Ozone Non-Rolling upgrades

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Labels: pull-request-available
• Target Version/s: 1.2.0
• Assignee: Aravindan Vijayan
• Reporter: Aravindan Vijayan
• Created: 01/Jun/20 19:23
• Updated: 17/Aug/22 15:51
• Resolved: 14/Jul/21 21:15

Description

Support for Non-rolling upgrades in Ozone.

Issue Links

• requires: RATIS-1305 Leader stuck in infinite install snapshot cycle when logs have been purged
• requires: RATIS-1194 SegmentedRaftLog syncWithSnapshot should not create a new open segment.
• requires: RATIS-1241 Leader unable to append logs to a recovering follower when its logs have been purged.
• requires: RATIS-1088 SegmentedRaftLogWorker#closeLogSegment should also roll the open segment maintained in the cache.
• requires: RATIS-1195 Log Entry Validation in appendEntry can fail after log purge.
• requires: RATIS-1216 State machine preAppend callback can throw exception without having leader to step down.
• links to: GitHub Pull Request #2121
• links to: GitHub Pull Request #2180
• links to: GitHub Pull Request #2390

Sub-Tasks

• Introduce Layout Feature interface in Ozone: Resolved
• Introduce OM layout version 'v0'.: Resolved
• Introduce SCM layout version 'v0'.: Resolved
• Add current layout version to OM Ratis Request: Resolved
• Implement Finalize command in Ozone Manager client.: Resolved
• Expose upgrade related state through JMX: Resolved
• Implement a factory for OM Requests that returns an instance based on layout version.: Resolved
• Implement Finalize command in Ozone Manager server.: Resolved
• Implement HDDS Version management using the LayoutVersionManager interface.: Resolved
• Add current HDDS layout version to Datanode heartbeat and registration.: Resolved
• Implement Datanode Finalization: Resolved
• SCM Finalize client command implementation.: Resolved
• Implement post-finalize SCM logic to allow nodes of only new version to participate in pipelines.: Resolved
• Schema Version field in Container metadata file should be backward compatible during read/write.: Resolved
• Add acceptance tests for upgrade, finalization and downgrade: Resolved
• Onboard HDDS-3869 into Layout version management: Resolved
• Revisit 'static' nature of OM Layout Version Manager.: Resolved
• Implement a "prepareForUpgrade" step that applies all committed transactions onto the OM state machine.: Resolved
• Add the current layout versions to DN - SCM proto payload.: Resolved
• SCM changes to process Layout Info in register request/response: Resolved
• Prepare for Upgrade step should purge the log after waiting for the last txn to be applied.: Resolved
• SCM changes to process Layout Info in heartbeat request/response: Resolved
• OM Layout Version Manager init throws silent CNF error in integration tests.: Resolved
• Investigate Acceptance test failure in Ozone Upgrade branch.: Resolved
• Add DataNode state and transitions for a node going through upgrade: Resolved
• Fix compilation issue in HDDS-3698-upgrade branch.: Resolved
• Verify that OM/SCM start fails when Software Layout Version < Metadata Layout Version: Resolved
• Ozone Manager Prepare for Upgrade/Downgrade design: Resolved
• Implement OM Prepare Request/Response: Resolved
• SCM restarts in the middle of the Upgrade should grace fully complete Upgrade: Resolved
• Add more unit tests for OM layout version manager.: Resolved
• Add a new OM admin command to submit the OMPrepareRequest.: Resolved
• Prepare client should check every OM individually for the prepared check based on Txn Id.: Resolved
• Add pre append gate and marker file to OM prepare state: Resolved
• Merge master into HDDS-3698-upgrade branch.: Resolved
• Fix issues in 'prepare' operation with one OM down.: Resolved
• Add an admin command to cancel "preparation" of an OM quorum.: Resolved
• Create OMCancelPrepareRequest and Response to cancel the prepared state of an OM.: Resolved
• Add Integration test for HDDS upgrade (happy path cases): Resolved
• Starting OM with the --upgrade flag should delete the prepare marker file.: Resolved
• Revisit LayoutFeature, and UpgradeAction related code: Resolved
• Fresh deploy of Ozone must use the highest layout version by default: Resolved
• Add read only command to get status of Finalization in OM & SCM.: Resolved
• Datanode unable to prepare itself for finalize.: Resolved
• SCM should go into "safe mode" until there is at least 1 pipeline to work with after finalization.: Resolved
• Attempting an SCM finalization after a failed / incomplete finalization.: Resolved
• Fix upgrade branch CI stability issues.: Resolved
• Add Layout version information to Recon datanode info API.: Resolved
• Layout version should be available in DB for an un-finalized OM to be finalized through a Ratis snapshot.: Resolved
• Validating HDDS upgrade in presence of failures: Resolved
• Onboard SCM HA as a new Layout Feature into upgrades.: Resolved
• Do not wait one heartbeat to move newly registered datanodes that match SCM's MLV from HEALTHY_READONLY to HEALTHY: Resolved
• NoSuchMethodException when wrapping RpcException on downgrade: Resolved
• Introduce First upgrade startup action and Pre-finalized state validation in Layout Feature.: Resolved
• SCM should not use pipelines with HEALTHY_READONLY datanodes: Resolved
• Upload upgrade design documentation to docs module.: Resolved
• Merge master with SCM HA changes into upgrade branch.: Resolved
• Add pre-finalize validation action for SCM HA: Resolved
• Attempt to remove state from *UpgradeFinalizer classes.: Resolved
• Track OM prepare intermittent integration test failure: Resolved
• Recover from failure during upgrade action: Resolved
• Adjust LICENSE and NOTICE files for the non-rolling upgrade branch: Resolved
• Upgrade related RPC calls should be allowed only for admins: Resolved
• Restructure the acceptance test groups (unsecure/secure/misc): Resolved
• Race condition in NodestateManager#addNode allows datanodes with lower MLV to be used in pipelines: Resolved
• Merge master into HDDS-3698-upgrade branch (04/30/21).: Resolved
• Do not fail SCM HA pre-finalize validation if SCM HA was already being used: Resolved
• Allow multiple OM request versions to be supported at same layout version (HDDS-2939).: Resolved
• Datanodes should always use MLV 0 when no VERSION file is present: Resolved
• Merge master branch at 12e2918 into upgrade branch: Resolved
• Remove getRequestType method from OM request classes.: Resolved
• Fix datanode capacity related race condition: Resolved
• Fix TestSCMNodeManager after merge of master at 1d8f972 into upgrade branch: Resolved