[SPARK-48082] Recover compatibility with Spark Connect client 3.5 <> Spark Connect server 4.0

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Done
• Affects Version/s: 4.0.0
• Component/s: Connect, PySpark
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 02/May/24 09:32
• Updated: 09/May/24 23:40
• Resolved: 09/May/24 23:40

Description

https://github.com/apache/spark/pull/46298#issuecomment-2087905857

There are test failures identified when you run Spark 3.5 Spark Connect client <> Spark Connect server 4.0.

They should ideally be compatible.

Sub-Tasks

• Backward compatibility test for Spark Connect: Resolved
• Skip tests being failed in client 3.5 <> server 4.0: Resolved
• ANSI enabled by default brings different results in the tests in 3.5 client <> 4.0 server: Resolved
• session.copyFromLocalToFs failure with 3.5 client <> 4.0 server: Resolved
• pyspark.ml.connect.evaluation not working in 3.5 client <> 4.0 server: Resolved
• Streaming exception catch failure in 3.5 client <> 4.0 server: Resolved
• Skip known behaviour change by SPARK-46122: Resolved
• Different Arrow versions in client and server: Resolved
• Python UDTF incompatibility in 3.5 client <> 4.0 server: Resolved
• Streaming query listener not working in 3.5 client <> 4.0 server: Resolved