Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-30367

De-duplicate InMemoryTableScan cached plans in SQL UI

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • 3.1.0
    • None
    • SQL, Web UI
    • None

    Description

      This is a follow-up JIRA for: https://issues.apache.org/jira/browse/SPARK-29431

      Currently with the change introduced by the JIRA this follows up on, duplicate subtrees of the query plan can be shown if multiple InMemoryTableScans read from the same persisted data:

      To prevent confusion, we should add an "InMemoryRelation" node that represents the persisted data being read from, and use it to de-duplicate shared plans like so:

      Attachments

        1. duplicated-imr.png
          115 kB
          Max Thompson
        2. deduplicated-imr.png
          75 kB
          Max Thompson

        Activity

          People

            Unassigned Unassigned
            maxthomp Max Thompson
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: