Trivial Description
A ShuffleExchange's coordinator can be null sometimes, and when we need to do a toString() on it, it'll go to ShuffleExchange.nodeName() and throw a MatchError there because of inexhaustive match – the match only handles Some and None, but doesn't handle null.
An example of this issue is when trying to inspect a Catalyst physical operator on the Executor side in an IDE:
child = {WholeStageCodegenExec@13881} Method threw 'scala.MatchError' exception. Cannot evaluate org.apache.spark.sql.execution.WholeStageCodegenExec.toString()
where this WholeStageCodegenExec transitively references a ShuffleExchange.
On the Executor side, this ShuffleExchange instance is deserialized from the data sent over from the Driver, and because the coordinator field is marked transient, it's not carried over to the Executor, that's why it can be null upon inspection.
Attachments
Issue Links
- relates to
-
SPARK-9858 Introduce an ExchangeCoordinator to estimate the number of post-shuffle partitions.
-
- Resolved
-
- links to
