[SPARK-50371] Introduce a shared plan representation

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: Connect, SQL
• Epic Link: Unified SQL Scala Interface
• Assignee: Unassigned
• Reporter: Herman van Hövell
• Created: 4 days ago 14:08
• Updated: 4 days ago 14:08

Description

Introduce a stable structured plan representation that can be used by both Connect and Classic. This can be used by explain, queryExecution, or a lightweight version of QueryExecutionListener.

This plan representation should support the following things:

• (Parsed/Logical/Pysical)Plans
  • Schema
  • Name
  • Children
• Expressions
  • Name
  • Children
• Subqueries
• Reuse of subqueries/plans.
• Metrics
• (optionally) Statistics

The plan representation must be serializable to and from proto.

For connect we may want to send it at the end of query execution. We do need to think about limits here.