[HUDI-1623] Solid completion time on timeline

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.0.0-beta1
• Component/s: Common Core
• Labels: pull-request-available
• Story Points: 5
• Epic Link: 1.X Format Changes
• Assignee: Danny Chen
• Reporter: Nishith Agarwal

Reviewers

Vinoth Chandar

• Created: 17/Feb/21 00:14
• Updated: 27/Nov/24 14:55
• Resolved: 13/Oct/23 10:40

Description

We suggest a new file naming for the completed metadata file:

${start_time}.${action}.${completion_time}

 

We also need a global Time Generator that can ensure the monotonical increasing generation of the timestamp, for example, maybe hold a mutex lock with the last generated timestamp backing up there. Say it may holds a lock L1. For each instant time generation, it needs guard from the lock.

 

Before creating the completed file, we also need a lock guard from L1.

 

Things need to note:
1. we only add completion timestamp to the completed metadata file;
2. we only add lock guard to the completed metadata file creation, not the whole commiting procedure;
3. for regular instant time generation, we also need a lock (that we should ship out by default)

Issue Links

• is related to: HUDI-5517 HoodieTimeline support filter instants by state transition time
• links to: GitHub Pull Request #9617
• links to: GitHub Pull Request #9972