[IMPALA-12293] OPTIMIZE statement to Compact Iceberg Tables

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: Frontend
• Labels: impala-iceberg
• Epic Link: Iceberg table maintenance tasks
• Epic Color: ghx-label-6
• Assignee: Noemi Pap-Takacs
• Reporter: Noemi Pap-Takacs
• Created: 17/Jul/23 13:00
• Updated: 25/Sep/24 11:32
• Resolved: 25/Sep/24 11:31

Description

A simple syntax to compact Iceberg tables. It executes the following tasks:

• compact small files
• rewrite partitions according to latest spec
• merge delete deltas

Syntax:

OPTIMIZE TABLE <table_name>
[ REWRITE DATA ]
[ ( { FILE_SIZE_THRESHOLD | MIN_INPUT_FILES } = <value> [, ... ] ) ]
[ WHERE <condition> ];

Limitations - OPTIMIZE TABLE can not be executed on the following tables:

• Non-Iceberg tables.
• Tables with complex types columns. Currently, Impala does not support writing complex types.
• If the 'write.format.default' is not Parquet. Impala can only write Parquet files.

Issue Links

• relates to: IMPALA-13382 OPTIMIZE could be more resistant to concurrent write operations

Sub-Tasks

• OPTIMIZE statement as an alias for INSERT OVERWRITE: Resolved
• Support partition evolution in OPTIMIZE statement: Resolved
• Filter files to OPTIMIZE based on file size: Resolved