[SPARK-34166] Fix flaky test in DecommissionWorkerSuite

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.1.0, 3.1.1, 3.2.0
• Fix Version/s: 3.1.1
• Component/s: Spark Core, Tests
• Assignee: XiDuo You
• Reporter: XiDuo You
• Created: 20/Jan/21 04:05
• Updated: 08/Feb/21 13:54
• Resolved: 20/Jan/21 05:27

Description

The test `decommission workers ensure that shuffle output is regenerated even with shuffle service` assumes it has two executor and both of two tasks can execute concurrently.

The two tasks will execute serially if there only one executor. The result is test is unexpceted. E.g. 

```

[info] 5 did not equal 4 Expected 4 tasks but got List(0:0:0:0-SUCCESS, 0:0:1:0-FAILED, 0:0:1:1-SUCCESS, 0:1:0:0-SUCCESS, 1:0:0:0-SUCCESS) (DecommissionWorkerSuite.scala:190)

```

Issue Links

• links to: [Github] Pull Request #31255 (ulysses-you)