[SPARK-19087] Numpy types fail to be casted to any other types

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 1.6.0
• Component/s: PySpark
• Labels: numpy type-converter

Environment

Red Hat 6.6, Spark 1.6.0, Python 2.7.10 (anaconda)

• Assignee: Unassigned
• Reporter: Ivan SPM
• Created: 05/Jan/17 17:19
• Updated: 12/Dec/22 18:11
• Resolved: 27/Jan/17 12:44

Description

An UDF cannot return a numpy type, it has to be one of the Python basic types. The error when a numpy type is returned is

TypeError: ufunc 'nan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

Issue Links

• duplicates: SPARK-12157 Support numpy types as return values of Python UDFs