[IMPALA-12426] SQL Interface to Completed Queries/DDLs/DMLs

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.4.0
• Component/s: Backend
• Labels: impala workload-management
• Epic Link: Workload Management
• Epic Color: ghx-label-1
• Assignee: Jason Fehr
• Reporter: Jason Fehr
• Created: 06/Sep/23 16:18
• Updated: 09/Oct/24 21:55
• Resolved: 01/Apr/24 21:44

Description

Implement a way of querying (via SQL) information about completed queries/ddls/dmls. Adds coordinator startup flags for users to specify that Impala will track completed queries in an internal table.

Impala will create and maintain an internal Iceberg table named "impala_query_log" in the "system database" that contains all completed queries. This table is automatically created at startup by each coordinator if it does not exist. Then, each completed query is queued in memory and flushed to the query history table either at a set interval (user specified number of minutes) or when a user specified number of completed queries are queued in memory. Partition this table by the hour of the query end time.

Data in this table must match the corresponding data in the query profile. Develop automated testing that asserts this requirement is true.

Don't write use, show, and set queries to this table.

Add the following metrics to the "impala-server" metrics group:

• Number of completed queries queued in memory waiting to be written to the table.
• Number of completed queries successfully written to the table.
• Number of attempts that failed to write completed queries to the table.
• Number of times completed queries were written at the regularly scheduled time.
• Number of times completed queries were written before the scheduled time because the max number of queued records was reached.

Issue Links

• causes: IMPALA-12660 TSAN error in ImpalaServer::QueryStateRecord::Init
• causes: IMPALA-12676 UBSAN intermittent int overflow on internal-server-test
• causes: IMPALA-12944 test_query_log_table_different_table failed in Ozone build
• requires: IMPALA-12624 Create a System Database for Impala

Sub-Tasks

• Remove Usage of "this->": Closed