[HDDS-505] OzoneManager HA

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Component/s: OM HA, Ozone Manager
• Labels: Triaged
• Target Version/s: 1.0.0
• Assignee: Hanisha Koneru
• Reporter: Hanisha Koneru
• Created: 18/Sep/18 22:08
• Updated: 09/Jun/20 17:20
• Resolved: 04/Jun/20 20:25

Description

OzoneManager can be a single point of failure in an Ozone cluster. We propose an HA implementation for OM using Ratis (Raft protocol).

Attached the design document for the proposed implementation.

Issue Links

• is duplicated by: HDDS-152 Support HA for Ozone Manager
• is duplicated by: HDDS-151 Add HA support for Ozone
• Parent Feature: HDDS-3723 OzoneManager HA (phase II)
• relates to: HDDS-2823 SCM HA Support

Sub-Tasks

• Start a Standalone Ratis Server on OM: Resolved
• Encapsulate all client to OM requests into one request message: Resolved
• Submit client request to OM Ratis server: Resolved
• Implement OzoneManager State Machine: Resolved
• Add support for configuring multiple OMs: Resolved
• Generate RaftGroupId from OMServiceID: Resolved
• Implement RetryProxy and FailoverProxy for OM client: Resolved
• Remove RaftClient from OM: Resolved
• Setup Failover Proxy Provider for OM client: Resolved
• Serve read requests directly from RocksDB: Resolved
• Add Tracing back to OzoneManagerProtocol: Resolved
• Provide docker-compose for OM HA: Resolved
• In OM HA AllocateBlock call where connecting to SCM from OM should not happen on Ratis: Resolved
• In OM HA OpenKey call Should happen only leader OM: Resolved
• In OM HA InitiateMultipartUpload call Should happen only leader OM: Resolved
• Implement Ratis Snapshots on OM: Resolved
• Convert all OM Volume related operations to HA model: Resolved
• Convert all OM Bucket related operations to HA model: Resolved
• Download RocksDB checkpoint from OM Leader to Follower: Resolved
• Convert all OM Key related operations to HA model: Resolved
• OzoneManager Cache: Resolved
• Implement DoubleBuffer in OzoneManager: Resolved
• Implement Bucket Write Requests to use Cache and DoubleBuffer: Resolved
• Add userName and IPAddress as part of OMRequest.: Resolved
• Implement AuditLogging for OM HA Bucket write requests: Resolved
• Implement updating lastAppliedIndex after buffer flush to OM DB.: Resolved
• Add metrics and AuditLogging for newly added OM HA methods: Resolved
• Implement Volume Write Requests to use Cache and DoubleBuffer: Resolved
• Create OMDoubleBuffer metrics: Resolved
• Implement Key Write Requests to use Cache and DoubleBuffer: Resolved
• Implement File CreateDirectory Request to use Cache and DoubleBuffer: Resolved
• Use ExecutorService in OzoneManagerStateMachine: Resolved
• Implement File CreateFile Request to use Cache and DoubleBuffer: Resolved
• Fix class hierarchy for KeyRequest and FileRequest classes.: Resolved
• Cleanup 2phase old HA code for Key requests.: Resolved
• Make OM KeyDeletingService compatible with HA model: Resolved
• Cleanup Volume Request 2 phase old code: Resolved
• Add Eviction policy for table cache: Resolved
• Implement S3 Create Bucket request to use Cache and DoubleBuffer: Resolved
• Fix numKeys metrics in OM HA: Resolved
• Implement S3 Delete Bucket request to use Cache and DoubleBuffer: Resolved
• Implement S3 Initiate MPU request to use Cache and DoubleBuffer: Resolved
• On installSnapshot notification from OM leader, download checkpoint and reload OM state: Resolved
• Fix OMVolumeSetQuota|OwnerRequest#validateAndUpdateCache return response.: Resolved
• Fix TestOzoneManagerHA and TestOzoneManagerSnapShotProvider: Resolved
• Implement S3 Commit MPU request to use Cache and DoubleBuffer: Resolved
• Implement S3 Abort MPU request to use Cache and DoubleBuffer: Resolved
• OzoneManagerDoubleBuffer#stop should wait for daemon thread to die: Resolved
• Merge code for HA and Non-HA OM requests for bucket: Resolved
• Fix TableCacheImpl cleanup logic: Resolved
• Make changes required for Non-HA to use new HA code in OM.: Resolved
• Implement S3 Complete MPU request to use Cache and DoubleBuffer: Resolved
• Fix failures in TestS3MultipartUploadAbortResponse: Resolved
• Convert all MPU related operations to HA model: Resolved
• Support volume acl operations for OM HA.: Resolved
• Support Bucket ACL operations for OM HA.: Resolved
• On OM reload/restart OmMetrics#numKeys should be updated: Resolved
• Support Key ACL operations for OM HA.: Resolved
• In OM HA getDelegation call Should happen only leader OM: Resolved
• Support Prefix ACL operations for OM HA.: Resolved
• Implement OM GetDelegationToken request to use Cache and DoubleBuffer: Resolved
• Implement OM CancelDelegationToken request to use Cache and DoubleBuffer: Resolved
• Implement OM RenewDelegationToken request to use Cache and DoubleBuffer: Resolved
• Implement GetS3Secret to use double buffer and cache.: Resolved
• Load Snapshot info when OM Ratis server starts: Resolved
• Implement default acls for bucket/volume/key for OM HA code: Resolved
• Handle Set DtService of token for OM HA: Resolved
• Make ozone fs shell command work with OM HA service ids: Resolved
• Add nullable annotation for OMResponse classes: Resolved
• Make ozone sh command work with OM HA service ids: Resolved
• Fix loadup cache for cache cleanup policy NEVER: Resolved
• Make OM Generic related configuration support HA style config: Resolved
• Handle Set DtService of token in S3Gateway for OM HA: Resolved
• Fix listBucket API: Resolved
• Fix listkeys API: Resolved
• Update OzoneServiceProvider in s3 gateway to handle OM ha: Resolved
• Add Volume check in KeyManager and File Operations: Resolved
• Fix listParts API: Resolved
• Fix listVolumes API: Resolved
• Run S3 test suite on OM HA cluster: Resolved
• Command line tool for OM Admin: Resolved
• Acceptance tests for OM HA: Resolved
• Send hostName also part of OMRequest: Resolved
• Fix logic of RetryPolicy in OzoneClientSideTranslatorPB: Resolved
• Add tests for incorrect OM HA config when node ID or RPC address is not configured: Resolved
• Fix listStatus API: Resolved
• Refactor OMFailoverProxyProvider#loadOMClientConfigs: Resolved
• Add support for Registered id as service identifier for CSR.: Resolved
• Add ServiceName support for getting Signed Cert.: Resolved
• Add ozone.om.internal.service.id to OM HA configuration: Resolved
• Fix listMultipartupload API: Resolved
• Remove RatisClient in OM HA: Resolved
• Merge OzoneManagerRequestHandler and OzoneManagerHARequestHandlerImpl: Resolved
• Merge OzoneClientFactory#getRpcClient functions: Resolved
• Ozone S3 CLI commands not working on HA cluster: Resolved
• Fix ApplyTransaction error handling in OzoneManagerStateMachine: Resolved
• Generate renewTime on OMLeader for GetDelegationToken: Resolved
• Make expiry of Delegation tokens to OM HA model.: Resolved
• HA failover attempt log level should be set to DEBUG: Resolved
• Allocating Blocks is not happening for Multipart upload part key in createMultipartKey call: Resolved
• Describe how ozoneManagerDoubleBuffer works in ascii art in code: Resolved
• Compare transactionID and updateID of Volume operations to avoid replaying transactions: Resolved
• Handle replay of KeyCreate requests: Resolved
• Handle replay of KeyDelete and KeyRename Requests: Resolved
• Handle Replay of AllocateBlock request: Resolved
• Add ObjectID and updateID to BucketInfo to avoid replaying transactions: Resolved
• Handle replay of KeyPurge Request: Resolved
• Consolidate ObjectID and UpdateID from Info objects into one class: Resolved
• Handle replay of KeyCommitRequest and DirectoryCreateRequest: Resolved
• Handle replay of S3 requests: Resolved
• Handle replay of OM Volume ACL requests: Resolved
• Handle replay of OM Key ACL requests: Resolved
• Handle replay of OM Prefix ACL requests: Resolved
• ACL checks should be done after acquiring lock: Resolved
• Delete replayed entry from OpenKeyTable during commit: Resolved
• Ozone S3 CLI path command not working on HA cluster: Resolved
• OM RpcClient fail with java.lang.IllegalArgumentException: Resolved
• Add unit tests for OMGetDelegationToken Request and Response: Resolved
• OM Client failover to next OM on NotLeaderException: Resolved
• Add wait time between client retries to OM: Resolved