[FLINK-28374] Some further improvements of blocking shuffle

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.16.0
• Component/s: Runtime / Network
• Assignee: Unassigned
• Reporter: Yingjie Cao
• Created: 04/Jul/22 08:16
• Updated: 14/Oct/22 10:09
• Resolved: 09/Aug/22 06:51

Description

This is an umbrella issue for sort-shuffle Improvements.

Issue Links

• relates to: FLINK-29641 SortMergeResultPartitionReadSchedulerTest.testCreateSubpartitionReader

Sub-Tasks

• Remove data flush in SortMergeResultPartition: Resolved
• Avoid notifying too frequently when recycling buffers for BatchShuffleReadBufferPool: Resolved
• Restrict the number of threads for sort-shuffle data read: Resolved
• Read a full buffer of data per file IO read request for sort-shuffle: Resolved
• Decrease the memory size per request for sort-shuffle data read from 8M to 4M: Resolved
• Produce one intermediate dataset for multiple consumers consuming the same data: Closed
• Introduce new compression algorithms of higher compression ratio: Resolved
• Select HashBasedDataBuffer and SortBasedDataBuffer dynamically based on the number of network buffers can be allocated for SortMergeResultPartition: Resolved
• Fix the bug that SortMergeResultPartitionReadScheduler may not read data sequentially: Resolved
• Store the number of bytes instead of the number of buffers in index entry for sort-shuffle: Resolved
• Remove the unused field in SortMergeSubpartitionReader: Resolved
• Extract header fields of Buffer into a BufferHeader class for blocking shuffle file IO: Resolved
• Allow multiple downstream consumer job vertices sharing the same intermediate dataset at scheduler side: Resolved
• Enlarge the max requested buffers for SortMergeResultPartitionReadScheduler: Resolved
• Sorting all unfinished readers in batches at one time in SortMergeResultPartitionReadScheduler: Resolved