[HUDI-7671] Make Hudi timeline backward compatible

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Component/s: core
• Labels: compatibility pull-request-available
• Epic Link: 1.0 Migration
• Assignee: Danny Chen
• Reporter: Danny Chen

Reviewers

Ethan Guo

• Created: 26/Apr/24 00:52
• Updated: 17/Jun/24 18:12
• Resolved: 17/Jun/24 18:12

Description

Since release 1.x, the timeline metadata file name is changed to include the completion time, we need to keep compatibility for 0.x branches/releases.

0.x meta file name pattern: ${instant_time}.action[.state]

1.x meta file name pattern: ${instant_time}_${completion_time}.action[.state].

In 1.x release, while decipheriing the Hudi instant from the metadata files, if there is no completion time, uses the file modification time as the completion time instead.

The modification time follows the OCC concurrency control semantics if the files were not moved around.

Caution that if the table is a MOR table and the files got moved in history from old folder to the current folder, the reader view may represent wong result set because the completion time are completely the same for all the alive instants.

Issue Links

• is duplicated by: HUDI-7857 Handle timeline instants without completion time
• links to: GitHub Pull Request #11443
• links to: GitHub Pull Request #11464