Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Incomplete
-
2.1.0
-
None
-
Important
Description
when run tpcds-q95, and set spark.sql.adaptive.enabled = true the physical plan firstly:
Sort
: +- Exchange(coordinator id: 1)
: +- Project***
: :-Sort **
: : +- Exchange(coordinator id: 2)
: : :- Project ***
: +- Sort
: : +- Exchange(coordinator id: 3)
spark.sql.exchange.reuse is opened, then physical plan will become below:
Sort
: +- Exchange(coordinator id: 1)
: +- Project***
: :-Sort **
: : +- Exchange(coordinator id: 2)
: : :- Project ***
: +- Sort
: : +- ReusedExchange Exchange(coordinator id: 2)
If spark.sql.adaptive.enabled = true, the code stack is : ShuffleExchange#doExecute --> postShuffleRDD function --> doEstimationIfNecessary . In this function,
assert(exchanges.length == numExchanges) will be error, as left side has only one element, but right is equal to 2.
If this is a bug of spark.sql.adaptive.enabled and exchange resue?