[HUDI-269] Provide ability to throttle DeltaStreamer sync runs

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Component/s: deltastreamer
• Labels: pull-request-available
• Assignee: Xing Pan
• Reporter: Balaji Varadarajan
• Created: 24/Sep/19 04:55
• Updated: 30/Sep/19 22:49
• Resolved: 30/Sep/19 22:49

Description

Copied from https://github.com/apache/incubator-hudi/issues/922

In some scenario in our cluster, we may want delta streamer to slow down a bit.
so it's nice to have a parameter to control the min sync interval of each sync in continuous mode.
this param is default to 0, so this should not affect current logic.

minor pr: #921

the main reason we want to slow it down is that aws s3 is charged by s3 get/put/list requests. we don't want to pay for too many requests for a really slow change table.

Issue Links

• links to: GitHub Pull Request #921