[SPARK-47289] Allow extensions to log extended information in explain plan

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Parth Chandra
• Reporter: Parth Chandra
• Created: 05/Mar/24 18:31
• Updated: 05/Apr/24 15:03
• Resolved: 05/Apr/24 02:35

Description

With session extensions, Spark planning can be extended to apply additional rules and modify the execution plan. If an extension replaces a node in the plan, the new node will be displayed in the plan. However, it is sometimes useful for extensions provided extended information to the end user to explain the impact of the extension. For instance an extension may automatically enable/disable some feature that it provides and can provide this extended information in the plan. 
The proposal is to optionally turn on extended plan information from extensions. Extensions can add additional planning information via a new interface that internally uses a new TreeNodeTag, say 'explainPlan'.

Issue Links

• links to: GitHub Pull Request #45488