[SPARK-42653] Artifact transfer from Scala/JVM client to Server

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Connect
• Epic Link: Spark Connect Scala Client - Basic
• Assignee: Venkata Sai Akhil Gudesa
• Reporter: Venkata Sai Akhil Gudesa
• Created: 02/Mar/23 15:37
• Updated: 22/May/23 04:55
• Resolved: 03/Mar/23 11:55

Description

In the decoupled client-server architecture of Spark Connect, a remote client may use a local JAR or a new class in their UDF that may not be present on the server. To handle these cases of missing "artifacts", we need to implement a mechanism to transfer artifacts from the client side over to the server side as per the protocol defined in https://github.com/apache/spark/pull/40147 

Issue Links

• links to: [Github] Pull Request #40256 (vicennial)
• links to: [Github] Pull Request #40256 (vicennial)
• links to: [Github] Pull Request #40267 (HyukjinKwon)