[SEDONA-663] Support spark connect in dataframe api

• Type: Improvement
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 1.7.0
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Sebastian Eckweiler
• Created: 14/Oct/24 06:17
• Updated: 24/Nov/24 08:16
• Resolved: 24/Nov/24 08:16

Description

In spark connect sessions, the `spark._jvm` attribute doesn't exist, which leads to errors when trying to invoke the python dataframe apis.

The apis in spark connect do custom function calls though, which could be used to implement the dataframe apis in spark-connect sessions.

Issue Links

• links to: GitHub Pull Request #1639