[IMPALA-4478] Account for Kudu client memory in MemTracker

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 2.8.0
• Fix Version/s: Impala 2.8.0
• Component/s: Backend
• Labels: kudu memory resource-management
• Target Version: Impala 2.8.0
• Assignee: Matthew Jacobs
• Reporter: Matthew Jacobs
• Created: 11/Nov/16 21:02
• Updated: 29/Nov/16 23:06
• Resolved: 29/Nov/16 23:06

Description

The Kudu client may consume non-trivial amounts of memory and it is not accounted for in the query MemTracker, so it may be possible for the process to run out of memory.

In particular, we need to consider the following in the KuduTableSink:

1. Buffer space for write ops to be sent, which is 100MB by default and is configurable via a flag.
2. Per-row errors observed by the client (before they are fetched and deleted by Impala). Each error contains a string and a copy of the row. The client API indicates that the error handling could overflow, i.e. that it is bounded, but the implementation does not yet limit the errors so this could be unbounded.

We need to also understand whether there are any non-negligible memory allocations in the KuduScanNode.