[IMPALA-6412] Memory issues with processing of incoming global runtime filters on coordinator

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.11.0
• Fix Version/s: Impala 4.0.0
• Component/s: Backend, Distributed Exec
• Labels: resource-management scalability
• Target Version: Product Backlog
• Epic Color: ghx-label-2
• Assignee: Unassigned
• Reporter: Tim Armstrong
• Created: 17/Jan/18 23:47
• Updated: 23/Dec/20 18:14
• Resolved: 23/Dec/20 18:14

Description

This is a tracking JIRA for runtime filter memory issues that occur when processing global runtime filters, which is currently done entirely on the coordinator. I'll link all related issues that I know about and we can close this issue once we're fully satisfied that the problem is resolved.

Currently the known issues include:

• All filters are processed on the coordinator, which quickly becomes a bottleneck
• We don't track the memory consumed by thrift strings in incoming RPCs

Issue Links

• depends upon: IMPALA-6153 Prevent Coordinator::UpdateFilter() running after query exec resources are released
• depends upon: IMPALA-3825 Distribute runtime filter aggregation across cluster
• depends upon: IMPALA-5486 Port control-plane parts of ImpalaInternalService to KRPC
• is related to: IMPALA-6596 Query failed with OOM on coordinator while remote fragments on other nodes continue to run
• relates to: IMPALA-3610 Track non-RPC memory from global runtime filters on the coordinator
• relates to: IMPALA-5519 Allocate fragment instance runtime filter memory from BufferPool