[SPARK-49248] Scala Client Parity with existing Dataset/DataFrame API

• Type: Epic
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: Connect
• Epic Name: Scala Client Parity with existing Dataset/DataFrame API
• Assignee: Pengfei Xu
• Reporter: Pengfei Xu
• Created: 15/Aug/24 14:50
• Updated: 28/Aug/24 02:52

Description

Implement missing APIs in the Scala Client to reach full parity with the existing Dataset interface.

Sub-Tasks

• [Connect] DF observe support for the scala client: Resolved
• Support df.MergeInto in Spark Connect: Resolved
• Implement df.mergeInto in PySpark: Resolved
• Schema evolution with `df.mergeInto` losing `when` clauses: Resolved
• Support UDAF.toColumn API in Spark Connect: Resolved
• Support UDAF in Spark Connect: Resolved
• MERGE with WITH SCHEMA EVOLUTION keywords: Resolved
• Clean up Mima rules for sql<>connect compatibility checks: Resolved
• (Tentitavely) Investigate Mima bug when checking Dataset APIs: Open
• Implement new Connect APIs in SQL Core: Resolved