[SPARK-42543] Specify protocol for UDF artifact transfer in JVM/Scala client

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Connect
• Epic Link: Spark Connect Scala Client - Basic
• Assignee: Venkata Sai Akhil Gudesa
• Reporter: Venkata Sai Akhil Gudesa
• Created: 23/Feb/23 21:28
• Updated: 01/Mar/23 16:39
• Resolved: 01/Mar/23 00:25

Description

An "artifact" is any file that may be used during the execution of a UDF.

In the decoupled client-server architecture of Spark Connect, a remote client may use a local JAR or a new class in their UDF that may not be present on the server. To handle these cases of missing "artifacts", a protocol for artifact transfer is needed to move the required artifacts from the client side over to the server side.

Issue Links

• links to: [Github] Pull Request #40147 (vicennial)
• links to: [Github] Pull Request #40147 (vicennial)