[HUDI-4615] Fix empty commits being made by deltastreamer with S3EventsSource when there is no data in SQS on starting a new pipeline

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.12.1
• Component/s: deltastreamer
• Labels: pull-request-available
• Story Points: 0.5
• Assignee: Vinish Reddy
• Reporter: sivabalan narayanan

Reviewers

sivabalan narayanan

• Created: 12/Aug/22 17:50
• Updated: 08/Sep/22 02:43
• Resolved: 08/Sep/22 02:43

Description

When we start a new deltastreamer with S3EventsSource, checkpoint is Option.empty(). After consumption from source, if there is no data, the source returns "val=0" as the checkpoint. So, deltastreamer assumes checkpoint has changed and makes an empty commit. This needs fixing. 

 

https://github.com/apache/hudi/blob/0d0a4152cfd362185066519ae926ac4513c7a152/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/S3EventsMetaSelector.java#L151

 

Issue Links

• links to: GitHub Pull Request #6387