[FLINK-28551] Store the number of bytes instead of the number of buffers in index entry for sort-shuffle

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.16.0
• Component/s: Runtime / Network
• Labels: pull-request-available
• Assignee: Yuxin Tan
• Reporter: Yingjie Cao
• Created: 14/Jul/22 09:47
• Updated: 22/Jul/22 05:43
• Resolved: 22/Jul/22 05:43

Description

Currently, in each index entry of sort-shuffle index file, one filed is the number of buffers in the current data region. The problem is that it is hard to know the data boundary before reading the file, to solve the problem, we can store the number of bytes instead of the number of buffers in index entry. Based on this change, we can do some optimization, for example, read larger size of data than a buffer for better sequential IO like what's mentioned in FLINK-28373.

Issue Links

• links to: GitHub Pull Request #20326