[SPARK-41758] Support Window.rowsBetween

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.4.0
• Component/s: Connect
• Assignee: Ruifeng Zheng
• Reporter: Sandeep Singh
• Created: 29/Dec/22 02:28
• Updated: 30/Dec/22 08:34
• Resolved: 29/Dec/22 12:13

Description

Doctest in pyspark.sql.connect.window.Window.rowsBetween fails with the error below:

File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/column.py", line 324, in pyspark.sql.connect.column.Column.over
Failed example:
    window = Window.partitionBy("name").orderBy("age")                 .rowsBetween(Window.unboundedPreceding, Window.currentRow)
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", line 1350, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.column.Column.over[1]>", line 1, in <module>
        window = Window.partitionBy("name").orderBy("age")                 .rowsBetween(Window.unboundedPreceding, Window.currentRow)
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/utils.py", line 346, in wrapped
        raise NotImplementedError()
    NotImplementedError

We should enable this back after fixing the issue in Spark Connect

Issue Links

• duplicates: SPARK-41292 Window-function support