[IMPALA-4522] Impalad crash under stress due to too many Kudu client threads

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 2.8.0
• Fix Version/s: Impala 2.8.0
• Component/s: Frontend
• Labels: crash kudu perf stress
• Target Version: Impala 2.8.0
• Assignee: Matthew Jacobs
• Reporter: Matthew Jacobs
• Created: 23/Nov/16 00:03
• Updated: 28/Nov/16 22:30
• Resolved: 28/Nov/16 22:30

Description

In stress testing on physical boxes (80 cores, 200gb ram) we discovered that the Kudu Java client creates a huge number of threads (2x the #cores) per Kudu client. Impala creates a Kudu client in catalog (table loading and DDL), during planning (fetching tablet locations for scans), and in the BE. The client created in planning is particularly problematic wrt the number of threads created, especially on machines with a large number of cores and under load. In the stress tests, this could result in the process crashing at some point when the JVM could not create more threads.

The attached hs_err.log shows this crash.

While we should explore sharing a single Kudu client (1 client needed per master), in the meantime we should reduce the number of threads the client creates, which is exposed in the Kudu API.