[HUDI-2683] Parallelize deleting archived hoodie commits

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 0.11.0
• Labels: pull-request-available sev:high
• Assignee: Unassigned
• Reporter: Yue Zhang
• Created: 04/Nov/21 02:19
• Updated: 22/Dec/21 06:21

Description

For now, hoodie will use 5s to delete 30 archived commits, even worse for bigger archive threshold like set archive.max_commits 100 or larger.

This is because of hoodie deleting archived commits in driver serially.

Sometimes, it is unacceptable for Spark Streaming jobs with second level batch interval.

We need to delete archived commits in parallel.

 

Issue Links

• links to: GitHub Pull Request #3920