[FLINK-31655] Adaptive Channel selection for partitioner

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Runtime / Task
• Assignee: tartarus
• Reporter: tartarus
• Created: 29/Mar/23 08:38
• Updated: 18/Jul/23 04:03

Description

In Flink, if the upstream and downstream operator parallelism is not the same, then by default the RebalancePartitioner will be used to select the target channel.

In our company, users often use flink to access redis, hbase or other rpc services, If some of the Operators are slow to return requests (for external service reasons), then because Rebalance/Rescale are Round-Robin the Channel selection policy, so the job is easy to backpressure.

Because the Rebalance/Rescale policy does not care which subtask the data is sent to downstream, so we expect Rebalance/Rescale to refer to the processing power of the downstream subtask when choosing a Channel.

Send more data to the free subtask, this ensures the best possible throughput of job!

 

 

 

Issue Links

• mentioned in: Page Loading...