[IMPALA-7115] Set a default THREAD_RESERVATION_LIMIT value

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 2.13.0, Impala 3.1.0
• Component/s: Backend
• Labels: resource-management
• Target Version: Impala 3.1.0
• Epic Color: ghx-label-5
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 01/Jun/18 22:46
• Updated: 22/Feb/19 03:55
• Resolved: 22/Jun/18 05:46

Description

As a follow on to IMPALA-6035, we should set a default value that actually will help protect again insanely complex queries.

Motivating discussion is here: https://gerrit.cloudera.org/#/c/10365/9/common/thrift/ImpalaInternalService.thrift

Tim Armstrong
1:11 PM

Dan suggested setting a default here. I started doing some experiments to see what our current practical limits are.

On stock Ubuntu 16.04 I start getting thread_resource_error at around 8000 reserved threads. I'm not sure that the config reflects what people would use on production systems so continuing to investigate.
Dan Hecht
1:31 PM

We could also consider choosing a default dynamically based on the OS's setting, if that's necessary.
Tim Armstrong
3:45 PM

I increased some of the configs (I think I was limited by /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max == 12288) and now it got oom-killed at ~26000 threads.

I think unfortunately there are a lot of different OS knobs that impact this and they seem to evolve over time, so it's probably not feasible with a reasonable amount of effort to get it working on all common Linux distros.

I was thinking ~5000, since 1000-2000 plan nodes is the most I've seen for a query running successfully in production.

Maybe I should do this in a follow-on change, since we probably also want to add a test query at or near this limit.

Issue Links

• is related to: IMPALA-6035 Add query option that rejects queries based on query complexity
• is related to: IMPALA-7136 Set a default THREAD_RESERVATION_AGGREGATE_LIMIT value

Sub-Tasks

• Doc THREAD_RESERVATION_LIMIT: Closed