[SPARK-41279] Feature parity: DataFrame API in Spark Connect

• Type: Umbrella
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 3.4.0
• Component/s: Connect
• Epic Link: Spark Connect
• Assignee: Ruifeng Zheng
• Reporter: Hyukjin Kwon
• Shepherd: Hyukjin Kwon
• Created: 28/Nov/22 01:25
• Updated: 29/Aug/23 04:15

Description

Implement DataFrame API in Spark Connect.

Issue Links

• split from: SPARK-39674 Initial protobuf definition for Spark Connect API

Sub-Tasks

• Better type errors when passing wrong parameters: In Progress
• Decouple plan transformation and validation on server side: Open
• Decouple handle command and send response on server side: Open
• Investigate the behavior difference in self-join: Open