[FLINK-29425] Hybrid full spilling strategy triggering spilling frequently

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.16.0
• Fix Version/s: 1.16.0
• Component/s: Runtime / Network
• Labels: pull-request-available
• Assignee: Weijie Guo
• Reporter: Weijie Guo
• Created: 27/Sep/22 04:21
• Updated: 28/Sep/22 08:50
• Resolved: 28/Sep/22 08:50

Description

In hybrid shuffle mode, we have an internal config option 'DEFAULT_FULL_STRATEGY_NUM_BUFFERS_TRIGGER_SPILLED' to control spilling frequency in the full spilling strategy. Unfortunately, the default value(10) is too small. As a result, frequent disk spilling calls are observed in the TPC-DS test, which seriously affects performance. When we increase the value, the query performance is improved significantly. We should set a more reasonable default value, or adopt an adaptive strategy to determine the spilling frequency.

Issue Links

• links to: GitHub Pull Request #20904