[SPARK-33814] Provide preferred locations for stateful operations without reported state store locations

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.2.0
• Component/s: Structured Streaming
• Assignee: L. C. Hsieh
• Reporter: L. C. Hsieh
• Created: 16/Dec/20 20:29
• Updated: 16/Dec/20 20:37

Description

Stateful operators in SS provides preferred locations on the previous batches if any. However, if there is no previous batch to follow, Spark possibly schedules stateful tasks in inefficient distribution. As stateful operations probably need to maintain large state stores, it is better we schedule stateful tasks across all executors.

Issue Links

• links to: [Github] Pull Request #30812 (viirya)