[HDDS-3816] Erasure Coding

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.3.0
• Component/s: OM, Ozone Client, Ozone Datanode, SCM
• Labels: EC pull-request-available
• Target Version/s: 1.3.0
• Assignee: Uma Maheswara Rao G
• Reporter: Uma Maheswara Rao G
• Created: 16/Jun/20 19:59
• Updated: 05/Jan/23 20:11
• Resolved: 20/Apr/22 22:08

Description

We propose to implement Erasure Coding in Apache Ozone to provide efficient storage. With EC in place, Ozone can provide same or better tolerance by giving 50% or more storage space savings.
In HDFS project, we already have native codecs(ISAL) and Java codecs implemented, we can leverage the same or similar codec design.

However, the critical part of EC data layout design is in-progress, we will post the design doc soon.

Also see HDDS-5351, which has a bunch of pre-requisites for the EC feature committed directly to the master branch.

Issue Links

• incorporates: HDDS-6462 Phase II : Erasure Coding Offline Recovery & Read/Write Improvements
• incorporates: HDDS-7731 Phase III: Erasure Coding followup tasks
• is related to: IMPALA-7003 Support erasure-coding in impala
• relates to: HDDS-3755 Storage-class support for Ozone
• requires: HDDS-5351 Erasure Coding Prerequisites
• requires: HDDS-6390 Facilities for compatibility - umbrella jira
• links to: GitHub Pull Request #2868
• links to: GitHub Pull Request #3301
• links to: Upgrade framework integration - problems and possible solutions

Sub-Tasks

• EC: Introduce the ReplicationConfig and modify the proto files: Closed
• EC: Persist replicationIndex on datanode side: Resolved
• Enhance SCMServerProtocol with using ReplicationConfig: Closed
• EC: allocateBlock API of OM->SCM should take the option EC policy/parameters, which can be used by SCM to select the pipeline provider: Closed
• EC: Add replicaIndex to the RPC protocols: Resolved
• EC: Introduce EC replication type: Resolved
• EC: Implement basic EC pipeline provider: Resolved
• EC: Create ECReplicationConfig on client side based on input string: Resolved
• EC: Implement the ECKeyOutputStream which should handle the EC mode writes: Resolved
• EC: Add ECReplicationConfig into KeyInfo proto: Resolved
• EC: Extend PipelineManager.createPipeline API to support excluded nodes: Resolved
• EC: Allow EC blocks to be requests from OM: Resolved
• EC: Add missing break in switch statement when requesting EC blocks: Resolved
• EC: Add configuration to set an EC container placement Policy: Resolved
• EC: allocateContainer should handle ec replication config: Resolved
• ContainerStateMap should handle ecReplication config map: Resolved
• EC: ECReplicationConfig should be immutable: Resolved
• EC: ContainerPlacementPolicyFactory#getPolicyInternal() should not be public: Resolved
• OMKeyRequest#createFileInfo should handle ECReplicationConfig: Resolved
• EC: Implement ECBlockInputStream to read a single EC Block Group.: Resolved
• EC: openKey/createFile should return whether the file created in EC mode: Resolved
• EC: Make ECReplicationConfig stored as bucket level attributes. (Example Encryption info): Resolved
• EC: Create a new as many racks as possible placement policy for EC: Resolved
• EC: Add padding and generate parity if the last stripe is not full: Resolved
• EC: Pipeline builder should copy replica Indexes from original pipeline: Resolved
• EC: commit key should consolidate and create one keyLocationInfo per blockGrp: Resolved
• EC: Provide replication config option from CLI when creating bucket.: Resolved
• EC: Resolve findbugs warnings after branch merge: Resolved
• EC: ECBlockOutputstream commitKey should create one keyLocationInfo per logical block: Resolved
• EC: Adapt KeyInputStream to read EC Blocks: Resolved
• EC: Add Codec and chunkSize to ECReplicationConfig: Resolved
• EC: ECKeyoutputStream should use codec and chunksize from ECReplicationConfig: Resolved
• EC: We should improve the behavior of BasicRootedOzoneClientAdapterImpl#getDefaultReplication with defaults configs moved to server.: Resolved
• EC: In BasicRootedOzoneClientAdapterImpl, Inherit bucket default replication config only in the case of EC.: Resolved
• EC: Remove hard coded chunksize and get from from ReplicationConfig: Resolved
• EC: Writing a large buffer to an EC file duplicates first chunk in block 1 and 2: Resolved
• EC: Fix TestRootedOzoneFileSystem.testBucketDefaultsShouldBeInheritedToFileForEC failure in branch: Resolved
• EC: ECKeyOutputStream#close fails if we write the partial chunk: Resolved
• EC: ECKeyOutputStream persists blocks in random order: Resolved
• EC: Implement seek on ECKeyInputStream: Resolved
• EC: Implement seek on ECBlockInputStream: Resolved
• EC: Adopt EC related utility from Hadoop source repository: Resolved
• EC: Refactor ECBlockOutputStreamEntry to accommodate all block group related ECBlockOuputStreams.: Resolved
• EC: Write should handle node failures.: Resolved
• EC: Integrate the Codec changes into EC Streams.: Resolved
• EC: Fix the compile issue in TestOzoneECClient (Due to concurrent commits): Resolved
• EC: Implement an Input Stream to reconstruct EC blocks ondemand: Resolved
• EC: Implement seek on ECBlockReconstructedStripeInputStream: Resolved
• EC: Review the current flush API and clean up: Resolved
• EC: ECBlockReconstructedStripeInputStream should handle block read failures and continue reading: Resolved
• EC: Fix the replication config handling in OMDirectoryCreateRequest#dirKeyInfoBuilderNoACL: Resolved
• EC: Fix TestOmMetrics in merge branch: Resolved
• EC: Create ECBlockReconstructedInputStream to wrap ECBlockReconstructedStripeInputStream: Resolved
• EC: Fix TestOzoneShellHA failures post master merge with EC branch: Resolved
• EC: Optimize ECBlockReconstructedStripeInputStream where there are no missing data indexes.: Resolved
• EC: Change CLI bucket default replication option name to "-type": Resolved
• EC: Track the failed servers to add into the excludeList when invoking allocateBlock: Resolved
• Centralize string based replication config validation via ReplicationConfigValidator: Resolved
• EC: Create ECInputStream wrapper to choose between reconstruction and normal reads.: Resolved
• EC: Provide set replicationConfig option to bucket: Resolved
• Fix ReplicationConfig related test failures that happened due to merging HDDS-5997 to the EC branch: Resolved
• EC: handleStripeFailure should retry: Resolved
• EC: ECBlockReconstructedStripeInputStream should read from blocks in parallel: Resolved
• EC: Provide CLI option to reset the bucket replication config: Resolved
• EC: HandleStripeFailure should not release the cachebuffers.: Resolved
• EC: Create reusable buffer pool shared by all EC input and output streams: Resolved
• EC: Client side exclude nodes list should expire after certain time period or based on the list size.: Resolved
• EC: Provide toString in ECReplicationConfig: Resolved
• EC: Pipeline creator should ignore creating pipelines for ZERO factor: Resolved
• EC: Review the TODOs in GRPC Xceiver client and fix them.: Resolved
• EC: Document the Ozone EC: Resolved
• EC: Introduce a gRPC client implementation for EC with really async WriteChunk and PutBlock: Resolved
• EC: Replication config from bucket should be refreshed in o3fs.: Resolved
• EC: put command should create EC key if bucket is EC: Resolved
• EC: Bucket does not display correct EC replication details: Resolved
• EC: Fix flakyness of tests around nodefailures: Resolved
• EC: putBlock should pass close flag true on end of block group/close file.: Resolved
• EC: Container Info command with json switch fails for EC containers: Resolved
• EC: Smoketest for ozone admin datanode expects exactly 3 nodes: Resolved
• EC: Datanode Chunk Validator fails on encountering EC pipeline: Resolved
• EC: Create EC acceptance test environment and some basic tests: Resolved
• EC: Replication Manager should skip EC Containers: Resolved
• EC: Parity blocks are incorrectly padded with zeros to the chunk size (#3043): Resolved
• EC: Add replica index to the output in the container info command: Resolved
• EC: Recon UI failing because of EC replication factor: Resolved
• EC: Read with stopped but not dead nodes gives IllegalStateException rather than InsufficientNodesException: Resolved
• EC: Pipelines for closed containers should contain correct replica indexes: Resolved
• EC: Fix unaligned stripe write failure due to length overflow.: Resolved
• EC: Freon ockg support EC write: Resolved
• EC: Apply fix for HDFS-16422 to the Ozone EC libraries: Resolved
• EC: Fix new checkstyle rule warnings in EC branch: Resolved
• EC: Make cluster-wide EC configuration take effect.: Resolved
• EC: Handle Replication Factor to consider EC Config in Recon UI: Resolved
• EC: Fix large write with multiple stripes upon stripe failure.: Resolved
• EC: Fix todo items in TestECKeyOutputStream: Resolved
• EC: Adapt java side native coder classes from hadoop.: Resolved
• EC: Update help strings for replication config: Resolved
• EC: Fix the race condition in TestECBlockReconstructedStripeInputStream: Resolved
• EC: Freon randomKeys EC key support: Resolved
• EC: Fix read big file failure with EC policy 10+4.: Resolved
• EC: PartialStripe failure handling logic is writing padding bytes also to DNs: Resolved
• EC: Do not throw NotImplementedException in flush(): Resolved
• EC: Refactor ECKeyOutputStream#write(): Resolved
• EC: Fix allocateBlockIfFull condition in ECKeyOutputStream#write(): Resolved
• EC: Calculate EC replication correctly when updating bucket usage: Resolved
• EC: Key Info command should not display legacy replication fields as they duplicate ReplicationConfig: Resolved
• EC: Exclude pipeline upon container close instead of exclude DNs.: Resolved
• EC: Automaticly switch to next OutputStream in ECBlockOutputStreamEntry: Resolved
• EC: Discard pre-allocated blocks to eliminate worthless retries.: Resolved
• EC: Review use of ReplicationConfig.getLegacyFactor() in the codebase: Resolved
• EC: add padding problem: Resolved
• EC: Ensure EC container usage is updated correctly when handling reports: Resolved
• EC: EC keys can't be created via S3 interfaces: Resolved
• EC: ListPipelines command should consider EC Containers: Resolved
• EC: Fix broken future chain and cleanup unnecessary validation function.: Resolved
• EC: Priory of replication config in Ozone FS: Resolved
• EC: Fix too many idle threads during reconstruct read.: Resolved
• EC: Avoid allocating buffers in EC Reconstruction Streams until first read: Resolved
• EC: OzoneManagerRequestHandler needs to handle ECReplicationConfig: Resolved
• EC: Container list command should allow filtering of EC containers: Resolved
• EC: Fix allocateBlock failure due to inaccurate excludedNodes check.: Resolved
• EC: Fix number of preAllocatedBlocks for EC keys: Resolved
• EC: [Refactor-1] Check isFullCell inside handleDataWrite: Resolved
• EC: Adjust requested size by EC DataNum in WritableECContainerProvider: Resolved
• EC: OmMultipartKeyInfo needs to handle ECReplicationConfig: Resolved
• EC: Reconstructed Input Streams should free resources after reading to end of block: Resolved
• EC: Improve exception message in ByteBufferEncodingState: Resolved
• EC: Handle ECReplicationConfig in initiateMultipartUpload API: Resolved
• EC: the offset is less than writeoffset: Resolved
• EC: Fix ISA-l load hadoop native lib UnsatisfiedLinkError: Resolved
• EC: OzoneMultipartUpload needs to handle ECReplicationConfig: Resolved
• EC: OzoneMultipartUploadPartListParts needs to handle ECReplicationConfig: Resolved
• EC: write when the datanode not enough: Resolved
• EC: Overwriting an EC key with a Ratis key fails: Resolved
• EC: Execute S3 acceptance tests with EC: Resolved
• EC: Unify replication-related CLI params: Resolved
• EC: [Forward compatibility issue] New client to older server could fail due to the unavailability for client default replication config: Resolved
• EC: scm CheckAndRecoverECContainer command: Resolved
• EC: Onboard EC into upgrade framework: Resolved