[IMPALA-12358] Skip scheduling runtime filter that unlikely to meet RUNTIME_FILTER_WAIT_TIME_MS constraint

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Frontend
• Labels: runtime-filters
• Epic Color: ghx-label-13
• Assignee: Unassigned
• Reporter: Riza Suminto
• Created: 09/Aug/23 23:11
• Updated: 09/Aug/23 23:11

Description

A scan node will wait for a runtime filter arrival for RUNTIME_FILTER_WAIT_TIME_MS. If the runtime filter is not arrived within this period, the scan node will stop waiting and start scanning. Late runtime filter can still be applied if the scanning has not finished yet upon filter arrival. However, planner should predict to some degree if a certain runtime filter is likely to be late. If it does, then skip scheduling such a filter.

The prediction can be based on either of:

• total volume of all scans from the build side of join.
• the distance between the join node and the furthest fragment in the build side direction.