[HUDI-6147] Retry Pending Compactions before ingestion in Deltastreamer

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.14.0
• Component/s: compaction, deltastreamer
• Labels: pull-request-available
• Assignee: Jonathan Vexler
• Reporter: Jonathan Vexler
• Created: 27/Apr/23 19:18
• Updated: 08/May/23 17:08
• Resolved: 08/May/23 17:08

Description

Currently, the timeline looks like this: The compaction is done after the deltacommit

drwxr-xr-x  2 jon  staff    64 Apr 27 15:15 archived
-rw-r--r--  1 jon  staff     0 Apr 27 15:15 20230427151550027.deltacommit.requested
drwxr-xr-x  4 jon  staff   128 Apr 27 15:15 metadata
-rw-r--r--  1 jon  staff   898 Apr 27 15:15 hoodie.properties
-rw-r--r--  1 jon  staff  2199 Apr 27 15:15 20230427151550027.deltacommit.inflight
-rw-r--r--  1 jon  staff  5269 Apr 27 15:15 20230427151550027.deltacommit
-rw-r--r--  1 jon  staff     0 Apr 27 15:15 20230427151558008.deltacommit.requested
-rw-r--r--  1 jon  staff  4365 Apr 27 15:15 20230427151558008.deltacommit.inflight
-rw-r--r--  1 jon  staff  6177 Apr 27 15:16 20230427151558008.deltacommit
-rw-r--r--  1 jon  staff  3211 Apr 27 15:16 20230427151602293.compaction.requested
-rw-r--r--  1 jon  staff     0 Apr 27 15:16 20230427151604955.deltacommit.requested
-rw-r--r--  1 jon  staff  4365 Apr 27 15:16 20230427151604955.deltacommit.inflight
-rw-r--r--  1 jon  staff  5977 Apr 27 15:16 20230427151604955.deltacommit
-rw-r--r--  1 jon  staff  1229 Apr 27 15:16 20230427151609434.rollback.requested
-rw-r--r--  1 jon  staff     0 Apr 27 15:16 20230427151609434.rollback.inflight
-rw-r--r--  1 jon  staff  1413 Apr 27 15:16 20230427151609434.rollback
-rw-r--r--  1 jon  staff     0 Apr 27 15:16 20230427151602293.compaction.inflight
-rw-r--r--  1 jon  staff  5271 Apr 27 15:16 20230427151602293.commit 

Issue Links

• links to: Apache Hudi Pull Request 8589
• links to: GitHub Pull Request #8589