[HUDI-3178] Metadata table compaction can include invalid updates from failed actions on dataset

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.10.1
• Labels: pull-request-available
• Story Points: 1
• Epic Link: Metadata Table for File Listing & Query Planning
• Assignee: sivabalan narayanan
• Reporter: Prashant Wason
• Created: 05/Jan/22 20:10
• Updated: 29/Jun/22 20:33
• Resolved: 26/Jan/22 13:34

Description

Metadata Table v2 performs an inline compaction once a deltacommit has been written.

Timeline:
(on dataset) t1.commit.requested
(on dataset) t1.commit.inflight
---- all parquet writes complete here, WriteStatus generated---
(on metadata table) t1.deltacommit.requested
(on metadata table) t1.deltacommit.inflight
(on metadata table) t1.deltacommit
---- deltcommit completed ----
(on metadata table) t1-001.compaction.requested
(on metadata table) t1-001.compaction.inflight
(on metadata table) t1-001.commit

If the t1.commit fails on the dataset then metadata table has already included information from the t1.commit in its base files which will be returned to readers. The metadata table reader logic only checks for deltacommits against completed instants on the dataset timeline and assumes a base file is always SANE.

Issue Links

• is duplicated by: HUDI-2869 Metadata bootstrapping should ignore data files from partial/inflight commits
• links to: GitHub Pull Request #4530