[FLINK-14845] Introduce data compression to blocking shuffle.

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.10.0
• Component/s: Runtime / Network
• Labels: pull-request-available
• Release Note: Now user can set taskmanager.network.blocking-shuffle.compression.enabled to true to enable data compression of blocking shuffle.
• Assignee: Yingjie Cao
• Reporter: Yingjie Cao
• Created: 19/Nov/19 03:26
• Updated: 22/Jan/20 13:41
• Resolved: 08/Dec/19 03:31

Description

Currently, blocking shuffle writer writes raw output data to disk without compression. For IO bounded scenario, this can be optimized by compressing the output data. It is better to introduce a compression mechanism and offer users a config option to let the user decide whether to compress the shuffle data. Actually, we hava implemented compression in our inner Flink version and  here are some key points:

1. Where to compress/decompress?

Compressing at upstream and decompressing at downstream.

2. Which thread do compress/decompress?

Task threads do compress/decompress.

3. Data compression granularity.

Per buffer.

4. How to handle that when data size become even bigger after compression?

Give up compression in this case and introduce an extra flag to identify if the data was compressed, that is, the output may be a mixture of compressed and uncompressed data.

 

We'd like to introduce blocking shuffle data compression to Flink if there are interests.

 

Issue Links

• relates to: FLINK-15308 Job failed when enable pipelined-shuffle.compression and numberOfTaskSlots > 1
• links to: GitHub Pull Request #10375