[FLINK-24898] Some further improvements of sort-shuffle - ASF JIRA

XML

Word

Printable

JSON

This is an umbrella issue including several further improvements of sort-shuffle.

is related to

FLINK-25636 FLIP-199: Change some default config values of blocking shuffle for better usability

relates to

FLINK-29641 SortMergeResultPartitionReadSchedulerTest.testCreateSubpartitionReader

1.	Throw PartitionNotFoundException if the partition file has been lost for blocking shuffle	Closed	Yuxin Tan
2.	Reset read buffer request timeout on buffer recycling for sort-shuffle	Closed	Yuxin Tan
3.	Shuffle data directories to make directory selection of different TaskManagers fairer	Closed	Yuxin Tan
4.	Make FileChannelManagerImpl#getNextPathNum select data directories fairly	Closed	Yuxin Tan
5.	Move buffer recycle in SortMergeSubpartitionReader out of lock to avoid deadlock	Closed	Yuxin Tan
6.	Remove the redundant lock in SortMergeResultPartition	Closed	Yuxin Tan
7.	Restrict the maximum number of buffers can be used per result partition for sort-shuffle	Closed	Yingjie Cao
8.	Reduce the maximum size of data output buffers per result partition for sort-shuffle	Closed	Yingjie Cao
9.	Adjust the maximum number of buffers can be used per result partition for data read of sort-shuffle	Closed	Yingjie Cao
10.	Adjust the generation of subpartition data storage order for sort-shuffle from random shuffle to random shift	Closed	Yingjie Cao
11.	Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance	Closed	Yingjie Cao
12.	Move read buffer allocation and output file creation to setup method for sort-shuffle result partition to avoid blocking main thread	Closed	Yuxin Tan
13.	Distribute the data read buffers more fairly among result partitions for sort-shuffle	Closed	Yingjie Cao