[IMPALA-3599] Impala should account for memory used by Thrift transport

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: Impala 2.6.0
• Component/s: Distributed Exec
• Labels: resource-management
• Target Version: Impala 2.9.0
• Assignee: Unassigned
• Reporter: Tim Armstrong
• Created: 23/May/16 23:02
• Updated: 27/Jan/17 21:59
• Resolved: 27/Jan/17 21:59

Description

Each thrift transport uses a non-trivial amount of memory (256kb of buffers) and Impala creates a lot of them. We should track this memory in a way that we can measure and understand how much memory the RPC stack is using.

I already prototyped a patch to track the amount of memory the thrift client cache is using, which is relatively straightforward. It will be a little hard to track the server memory, since we hand over control of the transport to thrift.