[SPARK-43797] Python User-defined Table Functions

• Type: Umbrella
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.5.0, 4.0.0
• Component/s: PySpark
• Assignee: Unassigned
• Reporter: Allison Wang
• Created: 25/May/23 17:40
• Updated: 08/Jun/24 17:54

Description

This is an umbrella ticket to support Python user-defined table functions.

Issue Links

• relates to: SPARK-39247 Support returning a table or set of rows in CREATE FUNCTION

Sub-Tasks

• Initial support for Python UDTFs: Resolved
• Support arrow-optimized Python UDTFs: Resolved
• Support Python UDTFs in Spark Connect: Resolved
• Support non-deterministic Python UDTFs: Resolved
• Support Python UDTFs with empty return values: Resolved
• Improve error messages for Python UDTFs with wrong number of outputs: Resolved
• Improve error messages for Python UDTF arrow type casts: Resolved
• Improve error messages for Python UDTF returning non iterable: Resolved
• Fix AssertionError when converting UDTF output to a complex type: Resolved
• Improve error messages for creating Python UDTFs with pickling errors: Resolved
• Disable arrow optimization by default for Python UDTFs: Resolved
• Improve error messages for regular Python UDTFs that return non-tuple values: Resolved
• Include the name of the UDTF in the error messages generated during the function execution: Open
• Support profiler for Python UDTFs: Open
• Refactor PythonUDTFRunner to send its return type separately: Resolved
• Support for UDTF to analyze in Python: Resolved
• Support Python UDTFs with empty schema: Resolved
• Query planning to support PARTITION BY and ORDER BY clause for table arguments: Resolved
• Add support for accumulator, broadcast, and Spark files in Python UDTF's analyze.: Resolved
• Set up memory limits for analyze in Python.: Resolved
• Add user guide for Python UDTFs: Resolved
• Improve the documentation for TABLE input arguments for UDTFs: Resolved
• Query execution to support PARTITION BY and ORDER BY clause for table arguments: Resolved
• Support named arguments in Python UDTF: Resolved
• Cache the pandas converter for Python UDTFs: Resolved
• Make Python UDTFs by default non-deterministic: Resolved
• Add SQL query test suites for Python UDTFs: Resolved
• Refactor Arrow Python UDTF: Resolved
• Improve Python UDTF arrow serializer performance: Open
• Add API in 'analyze' method to return partitioning/ordering expressions: Resolved
• Project out PARTITION BY expressions before 'eval' method consumes input rows: Resolved
• Add a new method `cleanup` in the UDTF interface: Resolved
• Add API for 'analyze' method to return a buffer to be consumed on each class creation: Resolved
• Refactor analyzeInPython function to make it reusable: Resolved
• Return useful error message if UDTF returns None for non-nullable column: Resolved
• Return specific error messages if UDTF 'analyze' method accepts or returns wrong values: Resolved
• Create API to stop consuming rows from the input table: Resolved
• Update API for 'analyze' partitioning/ordering columns to support general expressions: Resolved
• Create API to acquire execution memory for 'eval' and 'terminate' methods: Closed
• Create API for 'analyze' method to indicate subset of input table columns to select: Resolved
• Enforce that 'AnalyzeResult' 'orderBy' field is a list of pyspark.sql.functions.OrderingColumn: Resolved
• Create API for 'analyze' method to send input column(s) to output table unchanged: Resolved
• Create API for 'analyze' method to differentiate constant NULL arguments and other types of arguments: Resolved
• Support running Python UDTF 'analyze' method from Spark executors: Resolved
• Analyzer bug with multiple ORDER BY items for input table argument: Resolved
• [Bug] Partition indices are incorrect when UDTF analyze() uses both select and partitionColumns: Resolved