[FLINK-32870] Reading multiple small buffers by reading and slicing one large buffer for tiered storage

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.18.0
• Fix Version/s: 1.18.0
• Component/s: Runtime / Network
• Labels: pull-request-available
• Assignee: Yuxin Tan
• Reporter: Yuxin Tan
• Created: 15/Aug/23 10:30
• Updated: 12/Sep/23 03:43
• Resolved: 12/Sep/23 03:43

Description

Currently, when the file reader of tiered storage loads data from the disk file, it reads data in buffer granularity. Before compression, each buffer is 32K by default. After compressed, the size will become smaller (may less than 5K), which is pretty small for the network buffer and the file IO.
We should read multiple small buffers by reading and slicing one large buffer to decrease the buffer competition and the file IO, leading to better performance.

Issue Links

• links to: GitHub Pull Request #23255