[HUDI-7416] Add interface for StreamProfile to be used in StreamSync for reading and writing data.

• Type: New Feature
• Status: Closed
• Priority: Blocker
• Resolution: Done
• Fix Version/s: 0.15.0, 1.0.0
• Component/s: deltastreamer
• Labels: pull-request-available
• Story Points: 4
• Assignee: Vinish Reddy
• Reporter: Vinish Reddy
• Created: 16/Feb/24 09:49
• Updated: 09/Jun/24 22:45
• Resolved: 27/Feb/24 12:56

Description

Introducing a new class known as StreamProfile which contains details about how the next sync round in StreamSync should be consumed and written. For eg:

KafkaStreamProfile contains number of events to consume in this sync round.
S3StreamProfile contains the list of files to consume in this sync round
HudiIncrementalStreamProfile contains the beginInstant and endInstant commit times to consume in this sync round.
In future we can add the method for choosing the writeOperationType and indexType as well, for now {{streamProfile.getSourceSpecificContext() }}will be used to consume the data from the source.

Issue Links

• links to: GitHub Pull Request #10687
• links to: GitHub Pull Request #10701
• links to: GitHub Pull Request #10736