[SPARK-30762] Add dtype="float32" support to vector_to_array UDF

• Type: Story
• Status: Resolved
• Priority: Major
• Resolution: Done
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0, 3.1.0
• Component/s: MLlib, PySpark
• Target Version/s: 3.0.0, 3.1.0
• Assignee: Liang Zhang
• Reporter: Liang Zhang
• Created: 09/Feb/20 06:56
• Updated: 12/Dec/22 18:10
• Resolved: 13/Feb/20 16:02

Description

Previous PR: https://github.com/apache/spark/blob/master/python/pyspark/ml/functions.py

In the previous PR, we introduced a UDF to convert a column of MLlib Vecters to a column of lists in python (Seq in scala). Currently, all the floating numbers in a vector is converted to Double in scala. In this issue, we will add a parameter in the python function vector_to_array(col) that allows converting to Float (32bits) in scala, which would be mapped to a numpy array of dtype=float32.

 

Issue Links

• links to: GitHub Pull Request #27522