[IMPALA-8964] Increase runtime filter wait timeout for mt_dop

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Distributed Exec
• Labels: multithreading
• Epic Link: Multithreading upgrade path for large clusters
• Target Version: Product Backlog
• Epic Color: ghx-label-2
• Assignee: David Rorke
• Reporter: Tim Armstrong
• Created: 23/Sep/19 21:34
• Updated: 08/Feb/21 17:18

Description

When we enable joins for multithreaded plans, we should adjust the runtime filter wait time.

A large part of the motivation for the timeout was to allow parallelism between the different sides of the join - there was some concern that having a scan block indefinitely would effectively reduce the amount of parallelism that the plan executed with.

With multithreading, we want to get parallelism across multiple copies of the same fragment, rather than parallelism across different fragments. So this motivation no longer applies. Making the filter wait time unlimited would make query execution more predictable.

Issue Links

• depends upon: IMPALA-7984 Port UpdateFilter() and PublishFilter() to KRPC
• is blocked by: IMPALA-9019 Fix runtime filter bugs with mt_dop
• is blocked by: IMPALA-9065 Fix cancellation of RuntimeFilter::WaitForArrival()