[DRILL-5918] Allow manual configuration when using Runtime.getRuntime().availableProcessors()

• Type: Improvement
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 1.11.0
• Assignee: Karthikeyan Manivannan
• Reporter: Elijah Zupancic
• Created: 31/Oct/17 19:22
• Updated: 07/Mar/18 23:14

Description

Currently Drill auto-configures the number of threads in various thread pools based on the processor count:

https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/server/BootStrapContext.java#L110
https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/server/options/TypeValidators.java#L269
https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/rm/AbstractResourceManager.java#L55
https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/rm/DefaultResourceManager.java#L99

In a number of situations, this configuration is incorrect. In particular, the settings aren't correct when [running in a bare-metal container](https://docs.google.com/document/d/1WGPGiJtbJZPZBMSD9YHAAz-O4AlAET9qOnS_GeriZtE/edit?usp=sharing) because you don't necessarily have access to all of the compute shares or memory of the underlying host.

Ideally, the number of CPUs to use could be specified as a system parameter to Drill. This would allow users to configure Drill to their optimal settings.