[HUDI-3675] Add support to shutdown Deltastreamer gracefully on certain conditions w/ continuous mode

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Done
• Fix Version/s: 0.12.0
• Component/s: deltastreamer
• Labels: pull-request-available
• Story Points: 2
• Assignee: sivabalan narayanan
• Reporter: sivabalan narayanan

Reviewers

Sagar Sumit

• Created: 21/Mar/22 23:33
• Updated: 09/May/22 09:50
• Resolved: 09/May/22 09:50

Description

Here is the use-case: 

With continuous mode deltastreamer, the process have to be fully up consuming resources even if there is no data on the source. So, users may want to shutdown the process if there is no data to consume for some consecutive amount of times.

 

and later can restart a new pipeline after sometime.  something like, restart deltastreamer continuous mode after 1 hour of shutdown. 

 

But when started w/ continuous mode, do gracefull shudown if there is no new data for 5 consecutive times. 

 

Issue Links

• links to: GitHub Pull Request #5073