Description
AQE will optimize the logical plan once there is query stage finished. So for inner join, when two join side is all small to be the build side. The planner of converting logical plan to physical plan will select the build side as BuildRight if right side finished firstly or BuildLeft if left side finished firstly. In some case, when BuildRight or BuildLeft may introduce additional exchange to the parent node. The revert approach in OptimizeLocalShuffleReader rule may be too conservative, which revert all the local shuffle reader when introduce additional exchange not revert the local shuffle reader introduced shuffle. It may be expense to only revert the local shuffle reader introduced shuffle. The workaround is to apply the OptimizeLocalShuffleReader rule again when creating new query stage to further optimize the sub tree shuffle reader to local shuffle reader.
Attachments
Issue Links
- duplicates
-
SPARK-29538 Test failure: org.apache.spark.sql.execution.adaptive.AdaptiveQueryExecSuite.multiple joins
-
- Resolved
-
- links to