[HUDI-1575] Early detection by periodically checking last written commit & active markers

• Type: Epic
• Status: Open
• Priority: Blocker
• Resolution: Unresolved
• Fix Version/s: 0.13.0
• Component/s: writer-core
• Labels: pull-request-available
• Assignee: Yue Zhang
• Reporter: Nishith Agarwal
• Created: 02/Feb/21 23:42
• Updated: 26/Jul/22 13:49

Description

Check if there are more commits, try to do resolution based on its current markers, and abort for a currently running job to avoid using up resources and running a concurrent job if we already found a commit that happened in the meantime. This can give back so much of the cluster early and dramatically lower costs in the cloud.

Issue Links

• links to: GitHub Pull Request #6002
• links to: GitHub Pull Request #6003
• links to: GitHub Pull Request #6059
• links to: GitHub Pull Request #6133