[FLINK-30938] Release Testing: Verify FLINK-29766 Adaptive Batch Scheduler should also work with hybrid shuffle mode

• Type: Sub-task
• Status: Closed
• Priority: Blocker
• Resolution: Done
• Affects Version/s: 1.17.0
• Component/s: Runtime / Coordination
• Labels: release-testing
• Assignee: Yufan Sheng
• Reporter: Weijie Guo
• Created: 07/Feb/23 11:46
• Updated: 21/Feb/23 02:40
• Resolved: 21/Feb/23 02:40

Description

This ticket aims for verifying FLINK-29766: Adaptive Batch Scheduler should also work with hybrid shuffle mode.

More details about this feature and how to use it can be found in this documentation.

The verification is divided into two parts:

Part I: Verify hybrid shuffle can work with AdaptiveBatchScheduler

Write a simple Flink batch job using hybrid shuffle mode and submit this job. Note that in flink-1.17, AdaptiveBatchScheduler is the default scheduler for batch job, so you do not need other configuration.

Suppose your job's topology like source -> map -> sink, if your cluster have enough slots, you should find that source and map are running at the same time.

Part II: Verify hybrid shuffle can work with Speculative Execution

Write a Flink batch job using hybrid shuffle mode which has a subtask running much slower than others (e.g. sleep indefinitely if it runs on a certain host, the hostname can be retrieved via InetAddress.getLocalHost().getHostName(), or if its (subtaskIndex + attemptNumer) % 2 == 0)

Modify Flink configuration file to enable speculative execution and tune the configuration as you like

Submit the job. Checking the web UI, logs, metrics and produced result.

You should find that once a producer task's one subtask finished, all its consumer tasks can be scheduled in log.

Issue Links

• is caused by: FLINK-29766 AdaptiveBatchScheduler should also work with hybrid shuffle mode