[SPARK-49008] Propagate function signature in `DataFrame(...).transform(...)`

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.5.3
• Component/s: PySpark
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Nick Miller
• Created: 25/Jul/24 21:01
• Updated: 13/Nov/24 00:15

Description

Propagating the function signature for `func` in `DataFrame(...).transform(...)` would enable type checkers like `mypy` to enforce that `func` is being correctly called through `DataFrame(...).transform(...)`.

 

`ParamSpec` seems like a suitable approach for this:

P = ParamSpec("P")

...

def transform(
    self,
    func: Callable[Concatenate["DataFrame", P], "DataFrame"],
    *args: P.args,
    **kwargs: P.kwargs,
) -> "DataFrame":
    ...

Issue Links

• links to: GitHub Pull Request #47493