[IMPALA-3610] Track non-RPC memory from global runtime filters on the coordinator

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.5.0
• Fix Version/s: Impala 2.7.0
• Component/s: Backend
• Labels: runtime-filters scalability scale
• Target Version: Impala 2.7.0
• Assignee: Sailesh Mukil
• Reporter: Mostafa Mokhtar
• Created: 24/May/16 20:55
• Updated: 18/Jan/18 00:27
• Resolved: 18/Jan/18 00:26

Description

When running queries on a 1000x node cluster I noticed that the coordinator memory spikes due to global runtime filters, much of this memory is untracked and can easily cause the coordinator to the hit process memory limit.

NOTE: the fix for this issue did not fully solve the problem because it didn't track RPC payloads. See IMPALA-6412 for future work on the problem

Issue Links

• is related to: IMPALA-6412 Memory issues with processing of incoming global runtime filters on coordinator
• relates to: IMPALA-3701 Evaluate compressing Runtime filters to save coordinator network bandwidth