[IMPALA-13382] OPTIMIZE could be more resistant to concurrent write operations

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Catalog
• Labels: impala-iceberg
• Epic Color: ghx-label-6
• Assignee: Noemi Pap-Takacs
• Reporter: Zoltán Borók-Nagy
• Created: 13/Sep/24 19:41
• Updated: 25/Sep/24 11:32

Description

When there is a concurrent modification for a data file that is being replaced by the OPTIMIZE statement, we can get the following error:

Cannot commit, found new delete for replaced data file: ...

 Because of this we cannot commit OPTIMIZE, meaning all of its work is lost. Moreover, the newly written data files remain on storage as orphan files.

To avoid such conflicts, we could do the followings, before commiting OPTIMIZE:

1. Check if there is partition evolution involved in the file replacements. If so, let's just hope that the data files associated with deletes are not selected by OPTIMIZE, and jump straight to "commit OPTIMIZE". Otherwise:
2. Check if there are new snapshots since the base snapshot of the OPTIMIZE statement
3. If there are, then iterate over the snapshots
4. Collect the delete files (possibly via Snapshot.addedDeleteFiles())
5. Collect the set of partitions associated with delete files
6. Filter the file replacements by excluding the affected partitions (all have current partition spec)
   • We can also remove the newly written data files belonging to the affected partitions
7. Commit OPTIMIZE

We need to do this at partition-level granularity as we don't exactly know which new data files replace which old files. If partition evolution is involved, then we have absolutely no idea which new data files hold the data records coming from old partitioning.

Issue Links

• is related to: IMPALA-12293 OPTIMIZE statement to Compact Iceberg Tables