[IMPALA-691] Process mem limit should account for the JVM's memory usage

• Type: Improvement
• Status: Reopened
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: Impala 1.2.1, Impala 2.0, Impala 2.1, Impala 2.2, Impala 2.3.0
• Component/s: Backend
• Labels: incompatibility resource-management
• Target Version: Product Backlog
• Assignee: Unassigned
• Reporter: Skye Wanderman-Milne
• Created: 06/Dec/13 23:47
• Updated: 06/Jun/19 17:55

Description

The JVM doesn't appear to use malloc, so it's memory usage is not reported by tcmalloc and we do not count it in the process mem limit. I verified this by adding a large allocation in the FE, and noting that the total memory usage (virtual or resident) reported in /memz is not affected, but the virtual and resident memory usage reported by top is.

This is problematic especially because Impala caches table metadata in the FE (JVM) which can become quite big (few GBs) in extreme cases.

Workaround
As a workaround, we recommend reducing the process memory limit by 1-2GB to "reserve" memory for the JVM. How much memory you should reserve typically depends on the size of your catalog ( number of tables/partitions/columns/blocks etc.)

Sub-Tasks

• Add flag to count JVM memory against process limit: Resolved
• Limit JVM heap size in minicluster: Open
• Count JVM memory usage against process memory limit by default: Open