[HUDI-2579] Deltastreamer checkpoint metadata is not merged from previous commit instant

• Type: Task
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.10.0
• Component/s: deltastreamer, spark, writer-core
• Labels: pull-request-available
• Epic Link: Multi-writer & OCC
• Assignee: Dave Hagman
• Reporter: Dave Hagman

Reviewers

sivabalan narayanan

• Created: 19/Oct/21 13:46
• Updated: 21/Aug/22 08:11
• Resolved: 21/Aug/22 08:11

Description

Non-deltastreamer writers are supposed to copy over checkpoint metadata from previous checkpoints if the config hoodie.write.meta.key.prefixes = 'deltastreamer.checkpoint.key' is set. This does not happen which causes non-deltastreamer writers to corrupt the deltastreamer commits by essentially erasing the checkpoint metadata. 

Issue Links

• links to: GitHub Pull Request #3820