[HDDS-6462] Phase II : Erasure Coding Offline Recovery & Read/Write Improvements

• Type: Epic
• Status: Resolved
• Priority: Major
• Resolution: Implemented
• Fix Version/s: 1.4.0
• Assignee: Unassigned
• Reporter: Uma Maheswara Rao G
• Created: 16/Mar/22 18:17
• Updated: 11/Jul/23 15:12
• Resolved: 11/Jul/23 15:12

Description

This is an umbrella Jira for EC offline recovery work.

As part of Phase-I, we have finished the functionality of erasure coding write and reads as part of the Jira HDDS-3816. That being stabilized in a parallel effort. 

So, this Jira to start pending recovery work to finish end-end EC MVP.

Requirements in brief: 

1. The SCM to identify the lost containers and schedule for the reconstructions.
2. DNs to start reconstructing the containers upon the request from DN.
3. We can decide whether we create new RM at SCM for EC work or we just reuse existing one. Currently there are interest to start a new RM to start it clean as the existing one already complex enough.
4. DNs to figure out the blocks them self by interacting with multiple EC block containers as single EC container may not have full set of blocks. Either first container or parity containers should have full block set. 

I am splitting the offline recovery part of design from HDDS-3816 and post here soon.

Stay tuned for the updated doc.

We will also create new branch for this work in some time 

Issue Links

• is a parent of: HDDS-7661 Ratis Misreplication Handler
• is part of: HDDS-3816 Erasure Coding
• is related to: HDDS-8926 Umbrella for further EC improvements
• relates to: HDDS-7838 gRPC channel created for block input/output stream not shutdown properly
• split to: HDDS-7759 Improve Ozone Replication Manager

Sub-Tasks

• EC: Fix Datanode block file INCONSISTENCY during heavy load.: Resolved
• EC: EC pipeline records are not removed after close.: Resolved
• EC: Make ECBlockReconstructedStripeInputStream to be used by DNs as well: Resolved
• EC: [Refactor-2] Check and write parity cells inside handleDataWrite: Resolved
• EC: Scm CheckAndRecoverECContainer command: Resolved
• EC: Implement the EC Reconstruction Command with necessary information: Resolved
• Add a new replication manager and change the existing one to legacy: Resolved
• EC: Add BlockGroupLen info as part of PutBlock in EC Writes for helping in recovery.: Resolved
• EC: Refactor ECKeyOutputStream for better code reuse: Resolved
• EC: Add rpc for EC recovery in replication service: Resolved
• EC: ReplicationManager - create version of ContainerReplicaCounts applicable to EC: Resolved
• EC: Add EC pipeline minimum to MiniOzoneCluster: Resolved
• EC: Add the DN side Reconstruction Handler class.: Resolved
• EC: DN ability to create container in temp location and write blocks to it.: Resolved
• EC: Support ListBlock from CoordinatorDN: Resolved
• EC: ReplicationManager - create ContainerReplicaPendingOps class and integrate with ContainerManager: Resolved
• EC: ReplicationManager - make ContainerReplicaPendingOps into a SCM service: Resolved
• EC: PipelineStateMap#addPipeline should not have precondition checks post db updates: Resolved
• SCMContainerPlacementRackScatter should use original required node num to validate placement policy: Resolved
• EC: Fix datanode exclusion check in client: Resolved
• EC: DN ability to create RECOVERING containers for EC reconstruction.: Resolved
• EC: Add listBlock command MockDatanodeStorage for mocking in reconstruction work.: Resolved
• EC: Extend BlockReconstructedInputStreams to recover parity block buffers as well if missing: Resolved
• Need proper error message when "RATIS" replication-type is passed with EC codec: Resolved
• EC: getFileCheckSum should return null EC files until ECFileChecksum implemented.: Resolved
• EC: SCMContainerPlacementRackScatter#chooseDatanodes may choose less nodes than required in unknown cases.: Resolved
• EC: Validate the server default configuration on Ozone manager startup.: Resolved
• EC: Provide correct example for EC in ozone.server.default.replication: Resolved
• EC: Fix potential wrong replica read with over-replicated container.: Resolved
• EC: Standalone containers should not move to quasi closed: Resolved
• EC: ReplicationManager - create class to detect container health issues: Resolved
• EC: Implement the EC Reconstruction coordinator: Resolved
• EC: Analyze and add the block token support for ECReconstructionCoordinator: Resolved
• EC: Handle reconstructECContainersCommand in heartbeat: Resolved
• EC: ReplicationManager - collect under and over replicated containers: Resolved
• EC: EC Reconstruction Command count queues should be included in DN heartbeat: Resolved
• EC: ReplicationManager - Add class to handle under-replication and form a command for a datanode: Resolved
• EC: ReplicationManager - create class to form a replicate command for under replicated containers: Resolved
• EC: Ensure DatanodeAdminMonitor can handle EC containers during decommission: Resolved
• EC: Remove references to ContainerReplicaPendingOps in TestECContainerReplicaCount: Resolved
• EC: [Code Quality] Add more tests to get closure to 100% code coverage: Resolved
• EC: ReplicationManager - priortise under replicated containers: Resolved
• EC: Implement the Over-replication Handler: Resolved
• EC: Analyze and add putBlock even on non writing node in the case of partial single stripe.: Resolved
• EC: Attempt to cleanup the RECOVERING container when reconstruction failed at coordinator.: Resolved
• EC: Cleanup RECOVERING container on DN restarts.: Resolved
• EC: put key command with EC replication can use ReplicationConfig validator: Resolved
• EC: Add Test for RECOVERING container cleanup when failure.: Resolved
• EC: Skip the EC container for balancer: Resolved
• EC: CreateBucketHandler should use ReplicationConfig Validator: Resolved
• EC: Implement RECOVERING Container Scrubber.: Resolved
• EC: ReplicationManager - Logic to process the over replicated queues and assign work to DNs: Resolved
• EC: ReplicationManager - skip processing open containers: Resolved
• EC: Implement the Over replication Processor: Resolved
• EC: Add ec write channel: Resolved
• EC : support EC stripe shuffle: Resolved
• EC: Improve write performance by pipelining encode and flush: Resolved
• EC: Ensure replica index is maintained when replicating a container: Resolved
• EC: ReplicationManager - Over replication handler should set repIndex on delete cmds: Resolved
• NPE in ec.reconstruction.TokenHelper: Resolved
• Exception in Replication Monitor Thread: java.lang.IllegalArgumentException: Closed
• EC - ReplicationManager - handle maintenance only indexes in the under replication handler: Resolved
• EC: Prematurely re-throwed the exception in reconstruction cleanup loop.: Resolved
• EC: Add debug logging with exception info when stripe write failed: Resolved
• Support balancing EC container.: Resolved
• [Ozone EC] remove warnings and errors from console during online reconstruction of data.: Resolved
• EC: ReplicationManager - Track nodes already used when handing under replication: Resolved
• EC: Fix block deletion not allowed due to missing pipelineID: Resolved
• EC: ReplicationManager - UnderRep handler should handle duplicate indexes: Resolved
• EC: Fix offset Condition in ECKeyOutputStream: Resolved
• EC: DN reported Open EC container may not get closed if SCM container was already closed state?: Resolved
• EC: ReplicationManager - extend EC Container health check for mis-replication: Resolved
• EC: Add tests for erasure coding with MiniOzoneChaosCluster: Resolved
• EC: Handle maintenance replicas in ECUnderReplicationHandler: Resolved
• EC: decommission compatible offline recovery: Resolved
• EC: Define the value of Maintenance Redundancy for EC containers: Resolved
• EC: Schedule UnderReplicatedProcessor and OverReplicatedProcessor threads in RM instead of StorageContainerManager: Resolved
• EC: ReplicationManager - create handlers to perform various container checks: Resolved
• EC: ReplicationManager - Encapsulate the under and over rep queues into a queue object: Resolved
• Add Ratis tests for HealthCheck handlers of Replication Manager: Resolved
• EC: Add a Handler for CLOSING containers in Replication Manager: Resolved
• EC: Handle the placement policy check in ECUnderReplicationHandler: Resolved
• Add a handler for Quasi Closed containers to RM: Resolved
• EC: Change the placement policy interface to allow existing nodes to be specified.: Resolved
• EC: Fix tests for HealthCheck handlers of RM that use Replica Indexes for Ratis Containers: Resolved
• Erasure coding and encryption are not flagged on FileStatus: Resolved
• EC: EC Decode can fail when byteBuffer from elastic pool is larger than chunksize: Resolved
• Fixing exception handling in case of non positive replica index: Resolved
• EC: ECBlockReconstructedStripeInputStream should set initialized false on re-init: Resolved
• EC: ReplicationManager - move the empty container handling into RM from Legacy: Resolved
• EC: ReplicationManager - Implement ratis container health checker: Resolved
• EC: Close pipelines with unregistered nodes: Resolved
• EC: Add EC metrics: Resolved
• EC: Block allocation should not be stripped across the EC group: Resolved
• EC: ReplicationManager - implement deleting container handler: Resolved
• EC: ReplicationManager - LegacyReplicationManager should use the ContainerReplicaPendingOps service: Resolved
• EC: ReplicationManager - refactor Legacy RM to a container health detector: Resolved
• EC: Add a tool to schedule the EC Offline Reconstruction at any node.: Resolved
• EC: delete empty closed EC container: Resolved
• EC: ReplicationManager - Add relevant metrics to the various ReplicationManager classes: Resolved
• EC: ReplicationManager - handle UNHEALTHY replicas: Resolved
• EC: Add EC block checksum computer: Resolved
• EC: Notify ReplicationManager when a heartbeat updates datanode command counts: Resolved
• EC: ReplicationManager: Move Mis-Replicated into a separate unhealthy state: Resolved
• Cannot set bucket args when the volume has quota set: Resolved
• du command does not return correct disk consumed with replica for both ratis and EC: Resolved
• EC: Offline Recovery with simultaneous Over Replication & Under Replication: Resolved
• EC: Fix Reconstruction Issue with StaleRecoveringContainerScrubbingService: Resolved
• EC: ReplicationManager - refactor logic to send datanode commands into a central place: Resolved
• EC: Retry failed writes before rewrite to a new block group: Resolved
• EC: ReplicationManager - remove calls to ECHealthCheck from under and over replication processing: Resolved
• Eliminate duplicated config in LegacyReplicationManager: Resolved
• Add a handler for under replicated Ratis containers in RM: Resolved
• EC: Fix the NSSummaryEndpoint#getDiskUsage should be fixed for EC keys: Resolved
• Extend Placement Policy Interface to select mis-replicated replicas to copy: Resolved
• Add a handler for over replicated Ratis containers to RM: Resolved
• EC: Misreplication Handler changes for Placement Policy interface changes: Resolved
• Add subscription mechanism to ContainerReplicaPendingOps: Resolved
• ECUnderReplicationHandler does not consider pending adds when finding targets: Resolved
• EC: Refactor Unhealthy Replicated Processor: Resolved
• EC: Add debug logging to the Replication Manager check handlers: Resolved
• EC: Bug fix for calculating Misreplication Count: Resolved
• EC: ReplicationManager - merge mis-rep queue into under replicated queue: Resolved
• MisReplicationHandler does not consider QUASI_CLOSED replicas as sources: Resolved
• EC: "Missing" EC containers with some remaining replicas may block decommissioning: Resolved
• EC produces some unknown 1MB blocks without the control of deleting service: Resolved
• EC: SCM unregistered event handler for DatanodeCommandCountUpdated: Resolved
• EC: Enable balancer for EC containers.: Resolved
• EC: Increase the information in the RM sending command log message: Resolved
• EC metrics related to replication commands don't add up: Resolved
• EC: ECContainerReplicaCount should handle pending delete of unhealthy replicas: Resolved
• EC: Handle the placement policy satisfaction in HealthChecks handling: Resolved
• EC: Enhance datanode reconstruction log message: Resolved
• Placement Policy Interface changes to handle Overreplication: Resolved
• EC: ReplicationManager - Use placementPolicy.replicasToRemoveToFixOverreplication in EC Over replication handler: Resolved
• Refactor DefaultReplicationConfig: Resolved
• EC: GetChecksum for EC files can fail intermittently with IndexOutOfBounds exception: Resolved
• EC: Refactor ReplicationSupervisor to allow Replication and Reconstruction tasks: Resolved
• EC: Remove ECReconstructionSupervisor and send reconstruction commands to ReplicationSupervisor: Resolved
• EC: Add normal and low priority to replication supervisor and commands: Resolved
• EC: ECBlockInputStream should try spare replicas on error: Resolved
• EC: Change ContainerReplicaPendingOps to store deadline rather than scheduled time: Resolved
• [EC] Reconstruction is failing with IndexOutOfBoundsException: Resolved
• EC: ECPipelineProvider.createForRead should filter out dead replicas and sort replicas: Resolved
• EC: ECBlockReconstructedStripeInputStream should check for spare replicas before failing an index: Resolved
• EC: Add validation for EC chunk size: Resolved
• EC: SCM should throttle the reconstruction/replication tasks.: Resolved
• EC: Validate replication config at server-side: Resolved
• Test in ec/ozonefs.robot is not executed in CI: Resolved
• EC: Verify unrecoverable EC containers which are empty transition to deleting: Resolved
• EC: Offline reconstruction needs better logging: Resolved
• EC: Reconstruction could fail with orphan blocks.: Resolved
• TestContainerCommandsEC should close ECReconstructionCoordinator: Resolved
• Allow more EC pipelines based on number of volumes: Resolved
• Optimize getting open pipelines from pipelineManager: Resolved
• EC: WritableEcContainerProvider should dynamically adjust the open container groups: Resolved
• EC: Avoid O(n) array.remove(element) when filtering pipelines in WritableECContainerProvider: Resolved
• Defer non-critical partial EC reconstruction: Resolved
• EC: Avoid unbounded pipeline creation if all existing pipelines don't meet criteria: Resolved
• Use empty BufferPool for EC reconstruction: Resolved
• Split EC acceptance tests: Resolved
• Create acceptance test for offline recovery: Resolved
• UnsupportedOperationException when there are more replication tasks than limit: Resolved
• java.lang.IllegalArgumentException: ECContainerReconstructionThread: Resolved
• ECReconstructionCoordinator is not closed: Resolved