[IMPALA-9779] Unnecessarily reloading file metadata in some DDLs

• Type: Improvement
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: Impala 2.8.0, Impala 2.9.0, Impala 2.10.0, Impala 2.11.0, Impala 3.0, Impala 2.12.0, Impala 3.1.0, Impala 3.2.0, Impala 3.3.0, Impala 3.4.0
• Component/s: Catalog
• Epic Color: ghx-label-13
• Assignee: Unassigned
• Reporter: Quanlong Huang
• Created: 26/May/20 09:20
• Updated: 09/May/21 03:40

Description

Some DDLs are not modifying the actual table data. We don't need to reload file meta for them. These DDLs include:

• Compute (incremental) stats
• Drop stats
• Alter table set row format
• Alter table set file format

Code paths of them both call CatalogOpExecutor.bulkAlterPartitions(). The related partitions are marked as "dirty" anyway. Dirty partitions will be dropped and reloaded at the end of CatalogOpExecutor.alterTable(TAlterTableParams, TDdlExecResponse). See the details in HdfsTable.updatePartitionsFromHms().

We can consider not marking related partitions as "dirty" in these DDLs.

Issue Links

• is duplicated by: IMPALA-7910 COMPUTE STATS does an unnecessary REFRESH after writing to the Metastore
• relates to: IMPALA-6994 Avoid reloading a table's HMS data for file-only operations

Sub-Tasks

• Avoid unnecessarily loading metadata for compute stats DDL: Open