Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Incomplete
-
2.4.0
-
None
Description
Two major differences from ShuffleExchangeExec:
- The post-shuffle RDD should contain all the tasks, rather than reporting scheduler-level dependencies.
- The operator will have to manually register the continuous shuffle dependencies, since they're not reported to the scheduler.