[SPARK-49008] Propagate function signature in `DataFrame(...).transform(...)` - ASF JIRA

Attach files

Attach Screenshot

Add vote

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Open
Priority: Minor
Resolution: Unresolved
Affects Version/s: 3.5.3
Fix Version/s: None
Component/s: PySpark
Labels:
- pull-request-available

Description

Propagating the function signature for `func` in `DataFrame(...).transform(...)` would enable type checkers like `mypy` to enforce that `func` is being correctly called through `DataFrame(...).transform(...)`.

`ParamSpec` seems like a suitable approach for this:

P = ParamSpec("P")

...

def transform(
    self,
    func: Callable[Concatenate["DataFrame", P], "DataFrame"],
    *args: P.args,
    **kwargs: P.kwargs,
) -> "DataFrame":
    ...