[SPARK-48344] [M1] SQL Script execution

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Spark Core
• Labels: pull-request-available
• Assignee: Milan Dankovic
• Reporter: David Milicevic
• Created: 20/May/24 12:36
• Updated: 15 hours ago
• Resolved: 1 week ago 02:08

Description

Implement changes to sql() API to support SQL script execution:

• SparkSession changes
• sql() API changes - iterate through the script, but return only last DataFrame
• Spark Config flag to enable/disable SQL scripting in sql() API
• E2E testing

For more details, design doc can be found in parent Jira item.

Have a verbal agreement with Wenchen on the design, need to create design-doc and POC PR.

Issue Links

• links to: GitHub Pull Request #47403
• links to: GitHub Pull Request #47609
• links to: GitHub Pull Request #48879
• links to: GitHub Pull Request #48950