Details
-
Improvement
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
3.1.0
-
None
-
None
Description
This is a follow-up JIRA for: https://issues.apache.org/jira/browse/SPARK-29431
Currently with the change introduced by the JIRA this follows up on, duplicate subtrees of the query plan can be shown if multiple InMemoryTableScans read from the same persisted data:
To prevent confusion, we should add an "InMemoryRelation" node that represents the persisted data being read from, and use it to de-duplicate shared plans like so: