[HIVE-28341] Iceberg: Change Major QB Full Table Compaction to compact partition by partition

• Type: Task
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Component/s: Hive, Iceberg integration
• Labels: hive iceberg
• Assignee: Dmitriy Fingerman
• Reporter: Dmitriy Fingerman
• Created: 21/Jun/24 15:49
• Updated: 5 days ago 22:52

Description

Currently, Major compaction compacts a whole table in one step. If a table is partition and has a lot of data this operation can take a lot of time and it risks getting write conflicts at the commit stage. This can be improved to work partition by partition. Also, for each partition it will create one snapshot instead of 2 snapshots (truncate+IOW) created now when compacting the whole table in one step.