[HDDS-2823] SCM HA Support

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: SCM HA
• Labels: pull-request-available
• Target Version/s: 1.4.0
• Assignee: Li Cheng
• Reporter: Sammi Chen
• Created: 30/Dec/19 12:19
• Updated: 23/Jan/23 16:38
• Resolved: 23/Jan/23 16:37

Description

OM HA is close to feature complete now. It's time to support SCM HA, to make sure there is no SPoF in the system.

 

Design doc: https://docs.google.com/document/d/1vr_z6mQgtS1dtI0nANoJlzvF1oLV-AtnNJnxAgg69rM/edit?usp=sharing

Issue Links

• blocks: HDDS-4753 Onboard SCM HA Changes in to non-rolling upgrade framework
• is related to: HDDS-505 OzoneManager HA
• relates to: HDDS-5078 [SCM HA Security] NPE during secure SCM initialization with HA code updated to an already existing cluster
• relates to: HDDS-7823 SCM HA Phase 2
• relates to: HDDS-7331 Ozone PKI improvements
• links to: GitHub Pull Request #1606
• links to: GitHub Pull Request #1641
• links to: GitHub Pull Request #1715
• links to: GitHub Pull Request #1731
• links to: GitHub Pull Request #1747
• links to: GitHub Pull Request #1757
• links to: GitHub Pull Request #1854
• links to: GitHub Pull Request #1890
• links to: GitHub Pull Request #1908
• links to: GitHub Pull Request #1957
• links to: GitHub Pull Request #1972

Sub-Tasks

• Standalone SCM RatisServer: Resolved
• SCM StateMachine: Resolved
• Introduce generic SCMRatisRequest and SCMRatisResponse: Resolved
• SCM Invoke Handler for Ratis calls: Resolved
• Refactor configuration in SCMRatisServer to Java-based configuration: Resolved
• Handle AllocateContainer operation for HA: Resolved
• New PipelineManager interface to persist to RatisServer: Resolved
• Switch to PipelineStateManagerV2 and put PipelineFactory in PipelineManager: Resolved
• Introduce SCMStateMachineHandler marker interface: Resolved
• Add unit tests for new PipelineManager interface: Resolved
• Add unit test for SCMRatisResponse: Resolved
• Add unit test for SCMRatisRequest: Resolved
• Handle inner classes in SCMRatisRequest and SCMRatisResponse: Resolved
• decouple finalize and destroy pipeline: Resolved
• Implement container related operations in ContainerManagerImpl: Resolved
• Switch current pipeline interface to the new Replication based interface to write to Ratis: Resolved
• Add isLeader check for SCM state updates: Resolved
• remove the 1st edition of RatisServer of SCM HA which is copied from OM HA: Resolved
• update RATIS version from 1.0.0 to 1.1.0-85281b2-SNAPSHOT: Resolved
• RATIS ONE Pipeline is closed but not removed when a datanode goes stale: Resolved
• Pipeline is not removed when a datanode goes stale: Resolved
• Add failover proxy to SCM block protocol: Resolved
• enable SCM Raft Group based on config ozone.scm.names: Resolved
• CLI command to show current SCM leader and follower status: Resolved
• Switch to ContainerManagerV2: Resolved
• SCMBlockLocationFailoverProxyProvider should use ScmBlockLocationProtocolPB.class in RPC.setProtocolEngine: Resolved
• Handle PipelineAction and OpenPipline from DN to SCM: Resolved
• Make sure AllocateBlock can only be executed on leader SCM: Resolved
• Handle NodeReport from DN to SCMs: Resolved
• Handle events fired from PipelineManager to close container: Resolved
• Handle ContainerReport and IncrementalContainerReport: Resolved
• Replication can only be executed on leader: Resolved
• Use new ContainerManager in SCM: Resolved
• Add failover proxy for SCM container client: Resolved
• DN can distinguish SCMCommand from stale leader SCM: Resolved
• Fix CI and test failures after force push on 2020/10/26: Resolved
• Fix TestMiniOzoneHACluster.testGetOMLeader(): Resolved
• Add ReadWriteLock into PipelineStateManagerV2Impl to protect contentions between RaftServer and PipelineManager: Resolved
• Need throw exception to trigger FailoverProxyProvider of SCM client to work: Resolved
• Remove checkLeader in PipelineManager.: Resolved
• Add tests for replication annotation: Resolved
• SCM ServiceManager: Resolved
• Use getRoleInfoProto() in isLeader check: Resolved
• Handle stale leader issue: Resolved
• Add Snapshot into new SCMRatisServer and SCMStateMachine: Resolved
• SCM needs to replay RaftLog for recovery: Resolved
• BackgroundPipelineCreator can only serve leader: Resolved
• Implement Ratis Snapshots on SCM: Resolved
• DeleteBlock via Ratis in SCM HA: Resolved
• Load Snapshot info upon SCM Ratis starts: Resolved
• Allow Enabling Purge SCM Ratis log: Resolved
• Stop BackgroundPipelineCreator when PipelineManager is closed: Resolved
• SCMStateMachine::applyTransaction() should not invoke TransactionContext.getClientRequest(): Resolved
• Fix SCMHAManager#getPeerIdFromRoleInfo: Resolved
• Update pipeline db when pipeline state is changed: Resolved
• Avoid rewriting pipeline information during PipelineStateManagerV2Impl initialization: Resolved
• SCMContext Phase 1 - Raft Related Info: Resolved
• SCMContext: Resolved
• Handle potential data loss during ReplicationManager.handleOverReplicatedContainer(): Resolved
• Refactor SCMHAManager and SCMRatisServer with RaftServer.Division: Resolved
• Use OM style Configuration to initialize SCM HA: Resolved
• PipelineStateManagerV2Impl#removePipeline will remove pipeline from db in case of failure: Resolved
• acceptance test for SCM HA: Resolved
• Use suggestedLeader for SCM failover proxy performing failover: Resolved
• Bootstrap SCM HA Security: Resolved
• Use singe server raft cluster in MiniOzoneCluster.: Resolved
• Fix set configs in SCMHAConfigration: Resolved
• min/max election timeout of SCMRatisServer is not set properly.: Resolved
• Solve deadlock triggered by PipelineActionHandler.: Resolved
• Add term into SetNodeOperationalStateCommand.: Resolved
• Fix SCMHAManagerImpl#isLeader after RATIS-1227: Resolved
• Implement DB buffer in MockHAManager: Resolved
• Change default SCM snapshot frequency to a lower value: Resolved
• Ratis Snapshot should be loaded from the confg: Resolved
• Implement Distributed Sequence ID Generator: Closed
• replace scmID with clusterID for container and volume at Datanode side: Resolved
• Fix Recon after HDDS-4133: Resolved
• Should disallow log purge before installSnapshot is implemented: Resolved
• Backport updates from PipelineManager(V1): Open
• Handle pipeline reports: Open
• Handle ContainerAction and CloseContainer: Open
• Arrange Util classes for SCM HA: Open
• SCM CLI command towards certain IP: Open
• Design for SCM HA configuration: Open
• Provide docker-compose for SCM HA: Open
• Refactor out Ratis logic chain: Open
• SafeMode exit rule for all SCMs: Open
• Decommission can be only executed on leader: Open
• Design for Error/Exception handling in state update for container/pipeline V2: Open
• In ContainerStateManagerV2, modification of RocksDB should be consistent with that of memory state.: Open
• SCMBlockLocationFailoverProxyProvider should handle LeaderNotReadyException: Open
• Testing Infrastructure Random Failures: Open
• SCM HA needs handle the generation of clusterID and scmUuid in a decent way.: Open
• Add unit test to prove that datanode can handle term in SCMCommand properly: Open
• Handle inflight delete/add actions in ReplicationManager properly.: Open
• Implement InstallSnapshot for SCM HA: Open
• Merge OMTransactionInfo with SCMTransactionInfo: Resolved
• Disallow committing to DB by getCurrentBatchOperation(): Open
• Use applyTransactionSerial instead of applyTransaction: Resolved
• Support encode and decode ArrayList and Long: Resolved
• Replace UniqueID by the Distributed Sequence ID Generator: Resolved
• Bootstrap new SCM node: Resolved
• Admin command should take effect on all SCM instance: Resolved
• Add STOP state to SCMService.: Resolved
• activatePipeline/deactivatePipeline in PipelineManagerV2Impl should acquire lock before calling StateManager#updatePipelineState.: Resolved
• Implement increment count optimization in DeletedBlockLog V2: Resolved
• Add functionality to transfer Rocks db checkpoint from leader to follower: Resolved
• Add transactionId into deletingTxIDs when remove it from DB: Resolved
• Temporarily ignore failing Recon tests: Open
• Merge SCMRatisSnapshotInfo and OMRatisSnapshotInfo into a single class: Resolved
• Disable Prevote in Ratis in SCM HA by default: Resolved
• Fix findbugs issues after HDDS-2195: Resolved
• Fix TestContainerEndpoint after merging master to HDDS-2823.: Resolved
• Fix TestReconContainerManager after merge master to HDDS-2823: Resolved
• Fix misc acceptance test: List pipelines on unknown host: Resolved
• Add install checkpoint in SCMStateMachine: Resolved
• Integrate DeleteBlockLog with PartialTableCache: Resolved
• Add multiple SCM nodes to MiniOzoneCluster: Resolved
• [SCM HA Security] Implement generate SCM certificate: Resolved
• Use SCM service ID in SCMBlockClient and SCM Client: Resolved
• Implement scm --bootstrap command: Resolved
• Make SCM Generic config support HA Style: Resolved
• Move Ratis group creation to scm --init phase: Resolved
• Rename MiniOzoneHACluster to MiniOzoneOMHACluster: Resolved
• Use SCM service ID in finding SCM Datanode address.: Resolved
• Make changes required for SCM admin commands to work with SCM HA: Resolved
• Reopen replication/wait.robot added by HDDS-4834: Resolved
• Provide docker-compose for SCM HA: Resolved
• Datanode with scmID format should work with clusterID directory format: Resolved
• [SCM HA Security] Implement listCertificates based on role: Resolved
• [SCM HA Security] Add failover proxy to SCM Security Server Protocol: Resolved
• Make SCM ratis server spin up time during initialization configurable: Resolved
• Retry policy for SCM requests over ratis: Open
• Fix removing local SCM when submitting request to other SCM.: Resolved
• Fix and enable TestReconTasks: Resolved
• Fix and enable TestEndpoints.java: Resolved
• SCM Ratis enable/disable switch: Resolved
• Use PipelineManagerV2Impl in Recon and enable ignored Recon test cases.: Resolved
• Need a tool to upgrade current non-HA SCM node to single node HA cluster: Resolved
• [SCM HA Security] Create SCM Cert Client and change DefaultCA to allow self signed and intermediary: Resolved
• [SCM HA Security] Ozone services should be disabled in SCM HA enabled and security enabled cluster: Resolved
• Add SCM HA to Chaos tests: Resolved
• Add SCM to Ratis Log Parser: Open
• Support inline upgrade from containerId, delTxnId, localId to SequenceIdGenerator.: Resolved
• [SCM HA Security] Integrate CertClient: Resolved
• refactor code in SCMStateMachine.: Resolved
• NullPointerException during SCM init: Resolved
• [SCM HA Security] When Ratis is enabled, SCM secure cluster is not working: Resolved
• Provide example k8s files to run full HA Ozone: Resolved
• Return with exit code 0 in case of optional scm bootstrap/init: Resolved
• [SCM HA Security] Implement listCAs and getRootCA API: Resolved
• [SCM HA Security] Make CertStore DB updates for StoreValidateCertificate go via Ratis: Resolved
• [SCM HA Security] Handle leader changes during bootstrap: Resolved
• Fix flaky test TestSCMInstallSnapshotWithHA#testInstallCorruptedCheckpointFailure: Resolved
• Adapt admincli tests for SCM HA: Resolved
• Back-port HDDS-4911 (List container by container state) to ContainerManagerV2: Resolved
• Solve intellj warnings on DBTransactionBuffer.: Resolved
• Remove SequenceIdGenerator#StateManagerImpl: Resolved
• [SCM HA Security] Make storeValidCertificate method idempotent: Resolved
• [SCM HA Security] Make changes required for ratis enabled with new model of RootCA/subCA: Resolved
• [Doc] Add SCM HA Setup Doc: Resolved
• localId is not consistent across SCMs when setup a multi node SCM HA cluster.: Resolved
• SCM get roles command should provide Ratis Leader/Follower information.: Resolved
• SCM may not be able to know full port list of Datanode after Datanode is started.: Resolved
• Merge SCM HA configs to ScmConfigKeys: Resolved
• [SCM HA Security] Handle leader changes between SCMInfo and getSCMSigned Cert in OM: Resolved
• [SCM HA Security] Fix duration of sub-ca certs: Resolved
• [SCM HA Security] Make InterSCM grpc channel secure: Resolved
• [SCM HA Security] Remove code of not starting ozone services when Security is enabled on SCM HA cluster: Resolved
• [SCM HA Security] NPE during secure SCM initialization with HA code updated to an already existing cluster: Resolved
• Ensure failover to suggested leader if any for NotLeaderException: Resolved
• [SCM HA Security] Enable s3 test suite for ozone-secure-ha: Resolved
• make Decommission work under SCM HA.: Resolved
• Fix Install Snapshot Mechanism in SCMStateMachine: Resolved
• Divide snapshot related work into notifyInstallSnapshotFromLeader and reinitialize for SCMStateMachine.: Resolved
• If primordial SCM id is set, a non-HA cluster can not be initialized.: Resolved
• Use scm#checkLeader before processing client requests: Resolved
• Fix scm roles command if one of the host is unresolvable: Resolved
• For AccessControlException do not perform failover: Resolved
• ozone freon randomkeys failed after leader SCM node is down: Resolved
• Change default grpc and ratis ports for scm ha: Resolved
• Make admin check work for SCM HA cluster: Resolved
• SCM subsequent init failed when previous scm init failed: Resolved
• SCM UI should have leader/follower and Primordial SCM information: Resolved
• Fix Suggested leader in Client: Resolved
• Wait for ever to obtain CA list which is needed during OM/DN startup: Resolved
• SCM HA: Continuous PipelineNotFoundException seen in SCM log: Resolved
• Fix fall back of config in SCM HA Cluster: Resolved
• Handle unsecure cluster convert to secure cluster for SCM: Resolved
• Add reinitialize() for SequenceIdGenerator.: Resolved
• [SCM-HA] SCM start failed with PipelineNotFoundException: Resolved
• [SCM-HA] SCM start failed with PipelineNotFoundException: Resolved
• Add ratis snapshot retention policy for SCM HA: Open
• During bootstrap, always download checkpoint from leader SCM.: Open
• Handle NotLeaderException with Event Queue Handlers: Open
• Merge SCM HA Configuration: Open
• Add more tests for SCM Failover scenarios: Open
• [SCM HA Security] Make upgraded cluster to ratis enabled single node cluster: Open
• Backport updates from ContainerManager(V1): Open
• Add integration test for SequenceIdGen: Open
• Better handle the case that setting a trx that is earlier than latest trx in SCMDBTransactionBuffer: Open
• add new scm to SCM HA group: Open
• Support SCM HA in MiniOzoneHACluster: Open
• Use OM style config to construct RaftGroup and initialize Raft Servers: Resolved
• Prioritising SCM's in SCM HA setup: Open
• StorageContainerLocationProtocol api should throw SCMException: Open
• [SCM HA Security] generate certserialID in distributed sequence: Resolved
• remove scm from SCM HA group: Resolved
• Use MiniOzoneHAClusterImpl in TestDecommissionAndMaintenance.: Open
• For any IOexception from @Replicated method we should throw it: Resolved
• Add ratis metric for scm: Resolved
• use `Fileutils.move` instead of `Files.move` when installing snapshot: Resolved
• terminate om if statemachine is shut down by ratis: Resolved
• [Doc] Update OM HA Setup Doc: Resolved