[FLINK-19614] Further optimization of sort-merge based blocking shuffle - ASF JIRA

XML

Word

Printable

JSON

~~FLINK-19582~~ introduces a basic sort-merge based blocking shuffle implementation. We can further optimize it based on the approaches proposed in https://docs.google.com/document/d/1mpekX6aAHJhBsQ0pS9MxDiFQjHQIuaJH0GAQHh0GlJ0/edit?usp=sharing.

This is the umbrella ticket for the optimizations.

is related to

FLINK-21967 Add documentation on the operation of blocking result partition

relates to

FLINK-22153 Manually test the sort-merge blocking shuffle

1.	Implement shuffle data read scheduling for sort-merge blocking shuffle	Closed	Yingjie Cao
2.	Use managed memory to avoid direct memory OOM error for sort-merge shuffle (introduce a separated buffer pool)	Closed	Yingjie Cao
3.	Optimize data broadcast for sort-merge shuffle	Closed	Yingjie Cao
4.	Use region file mechanism for shuffle data reading before we switch to managed memory	Closed	Yingjie Cao
5.	Replace the 4M data writing cache of sort-merge shuffle with writev system call	Closed	Yingjie Cao
6.	Use heap memory instead of direct memory as index entry cache for sort-merge shuffle	Closed	Yingjie Cao
7.	Improve document and config description of sort-merge blocking shuffle	Closed	Yingjie Cao
8.	Fix wrong if condition in BufferReaderWriterUtil#writeBuffers	Closed	Yingjie Cao
9.	Enrich error message of read buffer request timeout exception	Closed	Unassigned
10.	Increase the default value of data writing cache size (not configurable) for sort-merge blocking shuffle	Closed	Unassigned