[RATIS-1938] Optimize stream friendly small file transfers

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Streaming
• Assignee: Unassigned
• Reporter: GuoHao
• Created: 20/Nov/23 04:30
• Updated: 20/Nov/23 21:09

Description

Currently, stream writing requires 3 steps
1. stream init
2. write data
3. close stream

For small files can we optimize this transfer process by combining these three steps into one and using a single rpc to merge the three steps?

 

message DataStreamPacketHeaderProto {
  enum Type {
    STREAM_HEADER = 0;
    STREAM_DATA = 1;
    STREAM_HEADER_DATA = 2;  // Add a new header, the init packet contains the data and is transmitted and then closed.   
  }

  enum Option {
    SYNC = 0;
    CLOSE = 1;
  }

  bytes clientId = 1;
  Type type = 2;
  uint64 streamId = 3;
  uint64 streamOffset = 4;
  uint64 dataLength = 5;
  repeated Option options = 6;
} 

 

I understand that https://issues.apache.org/jira/browse/RATIS-1157 can optimize the current process, but for small files it still keeps stream init.

Issue Links

• relates to: RATIS-1157 Buffer packets when the size of packets are too small