Details
-
Improvement
-
Status: In Progress
-
Major
-
Resolution: Unresolved
-
3.2.0
-
None
-
None
Description
Stateful operators in SS provides preferred locations on the previous batches if any. However, if there is no previous batch to follow, Spark possibly schedules stateful tasks in inefficient distribution. As stateful operations probably need to maintain large state stores, it is better we schedule stateful tasks across all executors.