[IMPALA-13012] Completed queries write fails regularly under heavy load

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 4.4.0
• Fix Version/s: Impala 4.4.0
• Component/s: Backend
• Epic Link: Workload Management
• Epic Color: ghx-label-9
• Assignee: Michael Smith
• Reporter: Michael Smith
• Created: 17/Apr/24 23:48
• Updated: 06/May/24 16:50
• Resolved: 26/Apr/24 23:46

Description

Under heavy test load (running EE tests), Impala regularly fails to write completed queries with errors like

W0411 19:11:07.764967 32713 workload-management.cc:435] failed to write completed queries table="sys.impala_query_log" record_count="10001"
W0411 19:11:07.764983 32713 workload-management.cc:437] AnalysisException: Exceeded the statement expression limit (250000)
Statement has 370039 expressions.

After a few attempts, it floods logs with an error for each query that could not be written

E0411 19:11:24.646953 32713 workload-management.cc:376] could not write completed query table="sys.impala_query_log" query_id="3142ceb1380b58e6:715b83d900000000"

This seems like poor default behavior. Options for addressing it:

1. Decrease the default for query_log_max_queued. Inserts are pretty constant at 37 expressions per entry. I'm not sure why that isn't 49, since that's the number of columns we have; maybe some fields are frequently omitted. I would cap query_log_max_queued to statement_expression_limit / number_of_columns ~ 5100.
2. Allow workload management to set statement_expression_limit higher using a similar formula. This may be relatively safe as the expressions are simple.
3. Ideally we would skip expression parsing and construct TExecRequest directly, but that's a much larger effort.

Issue Links

• relates to: IMPALA-13027 Directly create exec request for query log inserts