[FLINK-13792] source and sink support manual rate limit

• Type: Improvement
• Status: Open
• Priority: Not a Priority
• Resolution: Unresolved
• Affects Version/s: 1.8.1
• Component/s: Connectors / Common
• Labels: auto-deprioritized-major auto-deprioritized-minor
• Assignee: Unassigned
• Reporter: zzsmdfj
• Created: 20/Aug/19 07:25
• Updated: 23/Nov/21 10:43

Description

in current flink implement automatic flow control by back pressure, it is efficient for the most scene, but in some special scenario, do we need fine-grained flow control to avoid impact on other systems? For example: if i have window with days(a lot of datas), then do call ProcessWindowFunction when trigger, this will produce a lot of data to sink, if sink to message queue, it can have a huge impact to message queue. so if there is sink rate limiter, it is friendly to external system. for source rate limiter, it is appropriate for having window operator and accumulating a large amount of historical data.