[SPARK-28548] explain() shows wrong result for persisted DataFrames after some operations

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0
• Component/s: SQL
• Target Version/s: 3.0.0
• Assignee: Kousuke Saruta
• Reporter: Kousuke Saruta
• Created: 28/Jul/19 18:42
• Updated: 28/Dec/19 01:59

Description

After some operations against Datasets and then persist them, Dataset.explain shows wrong result.
One of those operations is explain() itself.
An example here.

val df = spark.range(10)
df.explain
df.persist
df.explain

Expected result is like as follows.

== Physical Plan ==
*(1) ColumnarToRow
+- InMemoryTableScan [id#7L]
      +- InMemoryRelation [id#7L], StorageLevel(disk, memory, deserialized, 1 replicas)
            +- *(1) Range (0, 10, step=1, splits=12)

But I got this.

== Physical Plan ==
*(1) Range (0, 10, step=1, splits=12)

Issue Links

• links to: GitHub Pull Request #25280