[SPARK-29994] Add WILDCARD task location

• Type: Task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Wei Xue
• Created: 22/Nov/19 03:49
• Updated: 21/Mar/20 00:10

Description

We have a "locality wait time" configuration that allows a task set to downgrade locality requirement after a certain time has passed. Yet, tasks all differ in penalty of locality miss. We propose a WILDCARD task location for tasks that do have a preferred location but would rather be assigned sooner than wait if the desired location is not available.

This requirement comes from LocalShuffledRowRDD. When the number of initial mappers of LocalShuffledRDD is smaller than the number of worker nodes, it can cause serious regressions if short-running tasks all wait on their preferred locations while they could have otherwise finished quickly on non-preferred locations too.

Issue Links

• links to: GitHub Pull Request #26633