[RATIS-979] Ratis streaming

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Component/s: Streaming
• Assignee: Tsz-wo Sze
• Reporter: Tsz-wo Sze
• Created: 17/Jun/20 18:50
• Updated: 04/Dec/23 21:10
• Resolved: 04/Dec/23 21:10

Description

In this JIRA, we design and implement Ratis Streaming with zero buffer copying and asynchronous event driven.

Issue Links

• is related to: RATIS-1542 Support TlsConf in netty streaming
• is related to: RATIS-1086 Streaming: support read requests
• is related to: HDDS-4454 Ozone Streaming Write Pipeline
• is related to: RATIS-1310 Use Ratis streaming to append log
• is related to: RATIS-1953 Ratis streaming follow up

Sub-Tasks

• Benchmarking Flatbuffers and Protobuffers for GRPC streaming: Resolved
• A simple benchmark achieving zero-copy semantics using Netty.: Resolved
• Define internal streaming APIs: Resolved
• Implement ratis streaming using netty: Resolved
• Implement ratis streaming using netty - Client code: Resolved
• Implement ratis streaming using netty - Server code: Resolved
• Update netty version - 4.1.50.Final: Resolved
• Update ratis-thirdparty version in order to use the latest netty: Resolved
• Refactor DataStream classes: Resolved
• Improve encoder/decoder by javadoc and error catch: Resolved
• Change NettyServerStreamRpc to write data to StateMachine: Resolved
• Netty stream server should forward the data to the other servers in the group: Resolved
• Create a transaction once the stream data is replicated to all servers: Resolved
• Support multiple groups in Streaming: Resolved
• Encode a RaftClientRequest as the head of a stream request: Resolved
• Add getDataStreamApi() to RaftClient: Resolved
• Implement RaftClient.getDataStreamApi(): Resolved
• DataStreamOutputImpl should use streamOffset instead of messageId: Resolved
• DataStreamReply should include byteWritten and isSuccess: Resolved
• DataStreamServerRpc should connect other peers automatically: Resolved
• Should check "bytewritten" from remote writes to peers to determine whether current stream write is successful: Resolved
• Refactor Netty streaming encoder and decoder: Resolved
• Add type for DataStream: Resolved
• Start NettyServerStreamRpc when create RaftServerProxy: Resolved
• Add SupportedDataStreamType.DISABLED: Resolved
• Fix DataStreamReply out of order: Resolved
• Change the FileStore example to use Streaming: Resolved
• Change NettyServerStreamRpc to use only one ConcurrentMap: Resolved
• NettyServerStreamRpc should handle duplicate headers: Resolved
• Add DataStreamType: Resolved
• Remove DataStreamClient.start() and DataStreamClientRpc.startClient(): Resolved
• Remove DataStreamServerRpc.start(): Resolved
• Support multiple streams: Resolved
• Support multiple DataStream clients: Resolved
• Use thenApplyAsync instead of thenApply: Resolved
• NettyServerStreamRpc should get stateMachine from RaftServer: Resolved
• Fix TestDataStream: Resolved
• Fix can not support more than 10 stream: Resolved
• Add a stream(RaftGroupId) method to DataStreamApi.: Resolved
• Move out the RPC related APIs from DataStreamOutput: Resolved
• Have a E2E test for Ratis streaming based on MiniRaftCluster: Resolved
• Primary and peer should use the same streamId: Resolved
• Primary and peer should use the same RaftClientRequest: Resolved
• Remove DataStreamApi.stream(RaftGroupId).: Resolved
• Initialize DataStreamServer after RaftServerProxy id is set: Resolved
• Add DataStreamRequestTypeProto: Resolved
• Rename StreamRequestTypeProto to MessageStreamRequestTypeProto: Resolved
• Add dataStreamAddress to RaftPeer: Resolved
• Do not create DataStreamOutput for non-primary server: Resolved
• Remove STREAM_CLOSE_FORWARD: Resolved
• Return exception of submitClientRequestAsync to client.: Resolved
• Default disable datastream when start a MiniRaftCluster: Resolved
• In NettyServerStreamRpc, the local/remote writes should only wait for the previous write: Resolved
• In NettyServerStreamRpc, each remote stream should not wait for its own previous write.: Resolved
• Return DataStreamException to client.: Resolved
• Move the DateStream requests handling code from NettyServerStreamRpc to a new class: Resolved
• Implement ChannelInboundHandler.exceptionCaught in NettyServerStreamRpc: Resolved
• Add sync flag when write stream data: Resolved
• Add a builder for DataStreamReplyByteBuffer: Resolved
• DataStreamClientImpl.closeAsync() should be idempotent: Resolved
• Enable testing clusters with Netty data stream and gRPC: Resolved
• Link DataStream with LogEntryProto: Resolved
• Move the utils from DataStreamBaseTest to a new file: Resolved
• Move the DataStream tests from TestDataStreamNetty to DataStreamClusterTests: Resolved
• Support WriteSmallFile by ratis streaming: Resolved
• Allow null for the stream parameter in StateMachine.DataApi.link: Resolved
• DataStreamApi/DataStreamOutput should expose stream close(): Resolved
• Rename StateMachineDataChannel to DataChannel: Resolved
• Test leader change when close stream: Resolved
• DataStreamOutput should support FileChannel#transferTo: Resolved
• Use RaftClient to submit request: Resolved
• Add a new API to DataStreamOutput for sending a File: Resolved
• Change the FileStore CLI to use Streaming.: Resolved
• Enhance FileStore steaming tests: Resolved
• Executor thread crashes on NPE: Resolved
• Remove streams from StreamMap when close stream: Resolved
• Provide a default implementation for DataStreamApi.stream(): Resolved
• Add sync when test filestore performance: Resolved
• FileStore stream to send small packets for MappedByteBuffer and NettyFileRegion: Resolved
• Stream small packet for NettyFileRegion and MappedByteBuffer: Resolved
• FileStore write file in parallel: Resolved
• The primary server should get the other peers from the current RaftConfiguration: Resolved
• FileStore support multi disk: Resolved
• Compare the performance between DataStreamApi and AsyncApi: Resolved
• FileStore client generate files in parallel: Resolved
• Save STREAM_CLOSE RPC call: Resolved
• Compare the performance between Ozone and DataStreamApi: Resolved
• Support network topology: Resolved
• Add route table in RaftConfiguration: Resolved
• FileStore support routing table: Resolved
• Move routingTable to RaftRpcRequestProto: Resolved
• Fix duplicated StreamMap.Key: Resolved
• Set default primary DataStreamServer in RaftClient.Builder: Resolved
• Close DataStreamClient when close RaftClient: Resolved
• Use ClientInvocationId as key instead of StreamMap.Key: Resolved
• Fix FileStore write failed because reply out of order: Resolved
• Resource leak when decoding DataStreamReplyByteBuffer: Resolved
• Validate RoutingTable before building it: Resolved
• Compare the performance between HDFS and DataStreamApi: Resolved
• Avoid parse proto for each packet: Resolved
• Support multi-client when transfer data between servers: Resolved
• Support multi-client in FileStore: Resolved
• Use async model for StreamInfo#applyToRemotes: Resolved
• Improve the used memory: Resolved
• Add an optional API to StateMachine.DataStream for providing an Executor: Resolved
• NettyClientStreamRpc#ChannelInboundHandler add exceptionCaught method: Resolved
• NettyClientStreamRpc#workerGroup Can cause too much nioEventLoopGroup object: Resolved
• make client NioEventLoopGroup size configurable.: Resolved
• NettyClientStreamRpc exception handling: Resolved
• Include commit infos in DataStreamReply: Resolved
• Move netty related code out from ratis-common: Resolved
• Cleanup NettyClientStreamRpc#replies empty queue: Resolved
• Fix log output in sendRequest: Resolved
• The file store client will hung when streaming writes.: Resolved
• Clean StreamMap when an exception occurs in DataStreamManagement#read: Resolved
• NettyServerStreamRpc get request resolving concurrency security: Resolved
• fix NettyClientStreamRpc.ChannelInboundHandler#clientInvocationId concurrency security problem: Resolved
• DataStreamManagement startTransaction error handler: Resolved
• OrderedStreamAsync error handler: Resolved
• make use CachedThreadPool configurable.: Resolved
• Add request timeout to ratis Streaming: Resolved
• Use the stream level SlidingWindow.client in OrderedStreamAsync: Resolved
• Change outstanding-requests.max default value.: Resolved
• NettyClientStreamRpc error reply handling: Resolved
• When DataStreamManagement#read an exception occurs, release ByteBuf: Resolved
• Streaming tests may fail with TimeoutIOException: Timeout 3000ms: Resolved
• NettyServerStreamRpc use EpollEventLoopGroup: Resolved
• Protocol message contained an invalid tag: Resolved
• Handle when there is an exception in NettyServerStreamRpc# getDataStreamOutput: Resolved
• OrderedStreamAsync uses a separate of send thread pool: Resolved
• Add metric to streaming: Resolved
• If the stream client is abnormally disconnected, reconnect the stream client: Resolved
• Make a new ratis snapshot version: Resolved
• Support reference-counted buffer in StateMachine.DataChannel: Resolved
• Discussion on improving the stability of streaming: Resolved
• NettyServerStreamRpc may throw NPE: Resolved
• The following assertion error occurs when transferring large files: Resolved
• Move the asyncRpcApi.sendForward(..) call to the client side.: Resolved
• Fix test failure from RATIS-1569: Resolved
• Add a ProxiesPool inner class in NettyServerStreamRpc.: Resolved
• Resolve the stream client concurrent reconnection problem: Resolved
• NettyServerStreamRpc.Proxies#getProxy NPE: Resolved
• DataStreamManagement#startTransaction prevent accidental assertion errors: Resolved
• Optional DataStreamManagement#startTransaction configuration: Resolved
• Buffer packets when the size of packets are too small: Resolved
• Save STREAM_HEADER RPC call: Resolved
• NettyServerStreamRpc should create client with TlsConf: Resolved
• Assert that the primary peers in DataStreamClient and RoutingTable are equal: Resolved
• Rewrite stream client reply queue.: Resolved
• Shared worker group in WorkerGroupGetter should be closed: Resolved