[HIVE-28515] Iceberg: Concurrent queries fail during commit with ValidationException

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.1.0, 4.0.1
• Component/s: Iceberg integration
• Labels: hive-4.0.1-merged hive-4.0.1-must pull-request-available
• Assignee: Ayush Saxena
• Reporter: Ayush Saxena
• Created: 10/Sep/24 06:55
• Updated: 24/Sep/24 02:15
• Resolved: 10/Sep/24 17:23

Description

Caused by: org.apache.iceberg.exceptions.ValidationException: Cannot commit, missing data files: [file:/Users/ayushsaxena/code/hive/iceberg/iceberg-handler/target/tmp/hive7073916777566968859/external/customers/data/00000-0-data-ayushsaxena_20240909232021_99fd025f-1e27-4541-ab3e-77c6f9905eb7-job_17259492220180_0001-6-00001.parquet]
        at org.apache.iceberg.MergingSnapshotProducer.validateDataFilesExist(MergingSnapshotProducer.java:751)
        at org.apache.iceberg.BaseRowDelta.validate(BaseRowDelta.java:116)
        at org.apache.iceberg.SnapshotProducer.apply(SnapshotProducer.java:233)
        at org.apache.iceberg.SnapshotProducer.lambda$commit$2(SnapshotProducer.java:384)
        at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413)
        at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:219)
        at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:203)
        at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196)
        at org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:382)
        at org.apache.iceberg.mr.hive.HiveIcebergOutputCommitter.commitWrite(HiveIcebergOutputCommitter.java:580)
        at org.apache.iceberg.mr.hive.HiveIcebergOutputCommitter.commitTable(HiveIcebergOutputCommitter.java:494)
        at org.apache.iceberg.mr.hive.HiveIcebergOutputCommitter.lambda$commitJobs$4(HiveIcebergOutputCommitter.java:291)

Queries failing with ValidationException during commit even with retry strategy configured with write_conflict

Issue Links

• links to: GitHub Pull Request #5438