[SPARK-40854] Change default serialization from 'broken' CSV to Spark DF JSON

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Connect
• Assignee: Martin Grund
• Reporter: Martin Grund
• Created: 20/Oct/22 08:12
• Updated: 12/Dec/22 18:10
• Resolved: 21/Oct/22 11:49

Description

Currently, Spark Connect uses a mediocre version of CSV serialization for the results from the server side without proper batching.

 

We should at least migrate to the existing Spark DF JSON serialization and batch the results.

Issue Links

• links to: [Github] Pull Request #38300 (grundprinzip)
• links to: [Github] Pull Request #38300 (grundprinzip)