[IMPALA-12412] Support partition evolution in OPTIMIZE statement

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: Frontend
• Labels: impala-iceberg
• Epic Color: ghx-label-3
• Assignee: Noemi Pap-Takacs
• Reporter: Noemi Pap-Takacs
• Created: 30/Aug/23 11:01
• Updated: 04/Mar/24 13:49
• Resolved: 04/Mar/24 13:49

Description

OPTIMIZE TABLE statement currently uses INSERT OVERWRITE to rewrite Iceberg tables. Therefore it inherits its limitations as well, such as the inability to rewrite tables with partition evolution.

This change aims to increase the support OPTIMIZE TABLE provides, by making it independent from INSERT statment. After the refactoring, OPTIMIZE TABLE  statement will be able to:

• rewrite all files in Iceberg tables according to the latest partition spec
• compact tables with partition evolution

This change also serves as a base for further improvements.

Syntax: OPTIMIZE TABLE <table_name>;