[IMPALA-4566] Kudu client glog contention can cause timeouts

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.8.0
• Fix Version/s: Impala 2.8.0
• Component/s: Backend
• Labels: kudu stress
• Target Version: Impala 2.8.0
• Assignee: Matthew Jacobs
• Reporter: Matthew Jacobs
• Created: 01/Dec/16 17:43
• Updated: 24/Jan/17 04:16
• Resolved: 09/Dec/16 23:43

Description

There appears to be significant contention in glog from the Kudu client threads under stressful workloads, we believe resulting in timeouts/failed queries.

Kudu JIRA: https://issues.apache.org/jira/browse/KUDU-695

Todd filed a glog upstream JIRA to address the contention. The Kudu client may need to be more careful with what is getting logged.

In Impala, we can mitigate some of this by setting the Kudu log level separately from the Impala log level, and offering the ability to disable the Kudu client logging completely.