[HUDI-1214] Need ability to set deltastreamer checkpoints when doing Spark datasource writes

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Fix Version/s: 0.11.0
• Component/s: spark
• Labels: sev:high user-support-issues
• Assignee: Trevorzhang
• Reporter: Balaji Varadarajan
• Created: 22/Aug/20 01:09
• Updated: 08/Dec/21 21:54
• Resolved: 08/Dec/21 21:54

Description

Such support is needed  for bootstrapping cases when users use spark write to do initial bootstrap and then subsequently use deltastreamer.

DeltaStreamer manages checkpoints inside hoodie commit files and expects checkpoints in previously committed metadata. Users are expected to pass checkpoint or initial checkpoint provider when performing bootstrap through deltastreamer. Such support is not present when doing bootstrap using Spark Datasource.