[SPARK-49568] Remove Self Type for Dataset

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Connect, SQL
• Labels: pull-request-available
• Epic Link: Unified SQL Scala Interface
• Assignee: Herman van Hövell
• Reporter: Herman van Hövell
• Created: 10/Sep/24 00:09
• Updated: 19/Sep/24 00:10
• Resolved: 19/Sep/24 00:10

Description

We are currently using a self type for Dataset. This makes it a bit easier (in Scala land) to deal with covariant parameters and return types.

The current approach is unfortunately a bit leaky. For example the self-type leaks into SparkSession and things like the Catalog. This is a bit distracting especially when you use these classes in java.

The proposal is to either remove the self-type and rely on overrides (which have to anyway), or to take another stab at using type members.

Issue Links

• links to: GitHub Pull Request #48146