Details
-
Umbrella
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
Description
We should adapt DataSkewPolicy to use compiler optimizer as a single point of optimization to align with our philosophy of DAG structure-aware optimization.
Currently, DataSkewRuntimePass optimizes PhysicalPlan instead of IR DAG in dynamic repartitioning of shuffle data. We need to fix this to optimize via IR DAG instead.