[SPARK-37431] pyspark.{ml, mllib}.linalg.Vector API

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.0
• Component/s: ML, MLlib, PySpark
• External issue URL: https://github.com/apache/spark/pull/34513#discussion_r744444312
• Assignee: Unassigned
• Reporter: Maciej Szymkiewicz
• Created: 21/Nov/21 13:50
• Updated: 21/Nov/21 13:51

Description

At the moment both Vector implementations have minimal API. mllib

class Vector(object):
    __UDT__ = ....
    def toArray(self): ...
    def asML(self): ...

ml

class Vector(object):
    __UDT__ = ...
    def toArray(self): ...

which doesn't cover actual API being used.

This causes typing issues, when we expect any Vector.

Issue Links

• relates to: SPARK-37234 Inline type hints for python/pyspark/mllib/stat/_statistics.py