[HDDS-4454] Ozone Streaming Write Pipeline

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: Ozone Client, Ozone Datanode
• Assignee: Tsz-wo Sze
• Reporter: Shashikant Banerjee
• Created: 12/Nov/20 05:42
• Updated: 11/Dec/23 19:08
• Resolved: 17/Dec/22 12:58

Description

https://issues.apache.org/jira/browse/RATIS-979 implements streaming semantics in Ratis.

Currently, ozone write pattern is bursty and involves multiple buffers copies as well multiple ratis log syncs in a block write. The idea of the Jira is to use a zero buffer copy based ratis streaming in ozone write path for better performance and resource utilisation. 

Issue Links

• is related to: HDDS-8691 Ozone Streaming Write Pipeline Phase 2
• relates to: HDDS-7651 Enable Ozone Streaming Write Pipeline by default
• relates to: RATIS-979 Ratis streaming
• relates to: HDDS-9903 EC data write as data streams

Sub-Tasks

• Implement stream method to ContainerStateMachine: Resolved
• Add link() method to ContainerStateMachine: Resolved
• Fix stream() and link() method in ContainerStateMachine: Resolved
• Client and server should support stream setup.: Resolved
• Add a new BlockOutputStream/KeyOutputStream to support streaming api: Resolved
• BlockDataStreamOutput: drop BufferPool and ChunkBuffer to avoid buffer copying: Resolved
• Change ByteBufStreamOutput to ByteBufferStreamOutput: Resolved
• Avoid unnecessary Bytebuffer conversions: Resolved
• Write exceptions occur after checksum is enabled: Resolved
• Add FileRegion API for Ozone Streaming: Closed
• Streaming supports writing in Pipline mode: Resolved
• Handle client retries on exception: Resolved
• Make raft.server.data-stream.client.pool.size configurable: Resolved
• Provide an Executor for each LocalStream in ContainerStateMachine: Resolved
• On the Datanode's ContainerStateMachine, the Stream and Link methods may get better performance by using chunkExecutors: Resolved
• Too many open files appear when an OzoneClient uses Streaming Write for a long time: Resolved
• Add XceiverClientRatis stream config: Resolved
• update the usage space of Containers in the stream write: Resolved
• OzoneBucket add the createMultipartStreamKey method: Resolved
• Add option to write files via streaming api in ofs and o3fs.: Resolved
• Support merge chunkInfos in client: Resolved
• Define a PutBlock/maxBuffer fixed boundary for streaming writes: Resolved
• Define a minimum packet size during streaming writes.: Resolved
• When releaseBuffers will get “Couldn 't find the required future”: Resolved
• Optimize parameters improve write performance: Resolved
• The 'Cannot allocate Memory 'problem occurs in client while test write: Resolved
• Fix incorrect computation of totalAckDataLength: Resolved
• Fix NPE in HDDS-6139: Resolved
• Add trace for streaming: Resolved
• Update ratis version to 2.3.0-94db58b-SNAPSHOT version: Resolved
• Add XceiverServerRatis stream config: Resolved
• Define a limit on the size of the retry bufferList .: Resolved
• Fix CheckStyle problem: Resolved
• Data Channel abstraction on datanode: Resolved
• Fix BlockDataStreamOutput#doFlushIfNeeded NPE: Resolved
• Streaming write support both pipeline model and star model: Resolved
• BlockDataStreamOutput support FlushDelay: Resolved
• Update Ratis version to 2.3.0-da5d868-SNAPSHOT: Resolved
• Setup TLS between the datanodes while streaming: Resolved
• Refactor KeyDataStreamOutput.: Resolved
• Buffer the PutBlockRequest at the end of the stream: Resolved
• Add option to createKey via streaming api via Freon: Resolved
• Fix ContainerStateMachine#applyTransaction assert error: Resolved
• Reduce the number of watch requests in StreamCommitWatcher: Resolved
• PutKeyHandler should not use streaming to put EC key.: Resolved
• Fix NPE in createStreamKey when replicationConfig is not set: Resolved
• Add BlockToken verification for streaming writes: Resolved
• Disable data steam by default: Resolved
• Add a createStreamKey method to OzoneBucket: Resolved
• Add documentation for the new Streaming Pipeline feature: Resolved
• [Ozone-Streaming] NPE in when creating a file with o3fs: Resolved
• Add a new acceptance test for Streaming Pipeline: Resolved