[SPARK-22239] User-defined window functions with pandas udf (unbounded window)

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.4.0
• Component/s: PySpark
• Assignee: Li Jin
• Reporter: Li Jin
• Created: 10/Oct/17 22:53
• Updated: 12/Dec/22 18:11
• Resolved: 13/Jun/18 01:11

Description

Window function is another place we can benefit from vectored udf and add another useful function to the pandas_udf suite.

Example usage (preliminary):

w = Window.partitionBy('id').rowsBetween(Window.unbounedPreceding, Window.unbounedFollowing)

@pandas_udf(DoubleType())
def mean_udf(v):
    return v.mean()

df.withColumn('v_mean', mean_udf(df.v1).over(window))

Issue Links

• links to: [Github] Pull Request #21082 (icexelloss)