[IMPALA-5737] Constrain and reduce memory requirements of minicluster and test processes

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: Impala 2.10.0
• Component/s: Infrastructure
• Epic Color: ghx-label-8
• Assignee: Tianyi Wang
• Reporter: Matthew Jacobs
• Created: 28/Jul/17 15:15
• Updated: 30/Apr/18 22:35

Description

The memory used by minicluster daemons and our test processes are not well constrained, which can lead to processes being killed by the OOM killer (e.g. in IMPALA-5733). Some processes have explicit memory limits (e.g. Kudu daemons have a 2gb limit), but Impala daemons consume an arbitrary amount (80% of memory). We need to also look at HDFS, HBase, Hive, etc. to make sure the limits are set appropriately. Further, it seems that we've never considered the memory used by test processes, e.g. python pytests, which spawn a process for each parallel worker - and this can add up especially in the minicluster stress tests.

We need to see how we can constrain memory requirements, and look for opportunities to reduce memory requirements. This will help avoid OOMs, it may allow us to run gerrit-verify-dryrun jobs with smaller instance types, and will help reduce the requirements for dev machines which will be good for the community.

Issue Links

• supercedes: IMPALA-5733 Kudu tservers seem to be unresponsive after TestKuduMemLimits