[SPARK-19427] UserDefinedFunction should support data types strings

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0, 2.1.0
• Fix Version/s: 2.2.0
• Component/s: PySpark, SQL
• Assignee: Maciej Szymkiewicz
• Reporter: Maciej Szymkiewicz
• Created: 01/Feb/17 17:55
• Updated: 13/Feb/17 18:38
• Resolved: 13/Feb/17 18:38

Description

PySpark SQL supports casting using data type strings.

from pyspark.sql.functions import col

col("foo").cast("integer")

It should be possible to do the same with udf / UserDefinedFunction:

from pyspark.sql import udf

udf(lambda x: x + 1, "integer")

Issue Links

• links to: [Github] Pull Request #16769 (zero323)