[HUDI-4021] Support deferring compaction when there is an inflight delta commit

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: compaction, multi-writer, table-service, writer-core
• Story Points: 3
• Assignee: sivabalan narayanan
• Reporter: sivabalan narayanan
• Created: 02/May/22 20:42
• Updated: 29/Mar/24 01:15

Description

in multi-writer scenario, if there is an inflight delta commit by writer1 and if writer2 i looking to schedule compaction, the delta commit might eventually fail. This may not be a good experience for end user. Think about regular writes are happening via deltastreamer and a separate async process is taking care of scheduling and executing compaction. Deltastreamer continuous mode might get aborted whenever such overstepping happens. We should add a support where in, compaction scheduling is deferred when an inflight delta commit is found in the timeline.