[SPARK-48075] Enforce type checking in from_avro and to_avro functions

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Structured Streaming
• Labels: pull-request-available
• Assignee: Fanyue Xia
• Reporter: Fanyue Xia
• Created: 01/May/24 20:13
• Updated: 02/May/24 05:39
• Resolved: 02/May/24 05:39

Description

Because types are not enforced in Python, users can pass in all sort of arguments for functions. This could lead to invoking wrong functions in spark connect.

If we perform type checking for arguments and output sensible errors when the type of arguments passed into the functions don’t match, we can give the user a better user experience

Issue Links

• links to: GitHub Pull Request #46324