[FLINK-33532] Move the serialization of ShuffleDescriptorGroup out of the RPC main thread - ASF JIRA

Agile Board

Rank to Top

Rank to Bottom

Attach files

Attach Screenshot

Bulk Copy Attachments

Bulk Move Attachments

Voters

Watch issue

Watchers

Convert to Issue

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Sub-task
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 1.19.0
Component/s: Runtime / Network
Labels:
- pull-request-available

Description

Currently, the serilization of ShuffleDescriptorGroup would happen in the RPC main thread even for light jobs. That would harm the QPS of OLAP session, e.g. the deserialization will take 3ms at TM side for each task of a 128*128 wordcount job. We propose to move it out of the rpc main thread. For heavy jobs, it still worth to cache the serialized value for memory efficiency. At the same time we need to make the serialization thread pool configurable.