[DRILL-5741] Automatically manage memory allocations during startup

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.11.0
• Fix Version/s: 1.13.0
• Component/s: Server
• Labels: ready-to-commit
• Docs Text: This would require a documentation change as well.
• Assignee: Kunal Khatua
• Reporter: Kunal Khatua
• Reviewer: Paul Rogers
• Created: 25/Aug/17 20:07
• Updated: 22/Mar/18 19:08
• Resolved: 24/Feb/18 16:40

Description

Currently, during startup, a Drillbit can be assigned large values for the following:

• Xmx (Heap)
• XX:MaxDirectMemorySize
• XX:ReservedCodeCacheSize
• XX:MaxPermSize

All of this, potentially, can exceed the available memory on a system when a Drillbit is under heavy load. It would be good to have the Drillbit ensure during startup itself that the cumulative value of these parameters does not exceed a pre-defined upper limit for the Drill process.

This JIRA is a proposal to allow for automatic configuration (based on configuration patterns observed in production Drill clusters). It leverages the capability of providing default/distribution (and user-specific) checks during Drill Startup from DRILL-6068.

The idea is to remove the need for a user to worry about managing the tuning parameters, by providing the optimal values. In addition, it also allows for the memory allocation to be implicitly managed by simply providing the Drill process with a single dimensional of total process memory (either in absolute values, or as a percentage of the total system memory), while auto-setup.sh provides the individual allocations.

This allocation is then partitioned into allocations for Heap and Direct Memory, with a small portion allocated for the Generated Java CodeCache as well. If any of the individual allocations are also specified (via distrib-env.sh or drill-env.sh), the remaining unspecified allocations are adjusted to stay within the limits of the total memory allocation.

The details of the proposal are here:
https://docs.google.com/spreadsheets/d/1N6VYlQFiPoTV4iD46XbkIrvEQesiGFUU9-GWXYsAPXs/edit#gid=0

For those unable to access the Google Document, PDFs are attached:

• Auto Mem Allocation Proposal - Computation Logic.pdf - Provides the equation used for computing the heap, direct and code cache allocations for a given input
• Auto Mem Allocation Proposal - Scenarios.pdf - Describes the various inputs, and their expected allocations

The variables that are (optionally) defined (in memory, distrib-env.sh or drill-env.sh ) are:

• DRILLBIT_MAX_PROC_MEM : Total Process Memory
• DRILL_HEAP : JVM Max Heap Size
• DRILL_MAX_DIRECT_MEMORY : JVM Max Direct Memory Size
• DRILLBIT_CODE_CACHE_SIZE : JVM Code Cache Size

Note: With JDK8, MaxPermSize is no longer supported, so we do not account for this any more, and will unset the variable if JDK8 or higher is detected.

Issue Links

• requires: DRILL-5560 Create configuration file for distribution specific configuration
• requires: DRILL-6068 Drill should support user/distribution specific configuration checks during startup
• links to: GitHub Pull Request #1082