[SPARK-26591] Scalar Pandas UDF fails with 'illegal hardware instruction' in a certain environment

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Feedback Received
• Affects Version/s: 2.4.0
• Component/s: PySpark

Environment

Python 3.6.7

Pyspark 2.4.0

OS:

Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

CPU:

 

Dual core AMD Athlon II P360 (-MCP-) cache: 1024 KB
clock speeds: max: 2300 MHz 1: 1700 MHz 2: 1700 MHz

 

 

• Assignee: Unassigned
• Reporter: Elchin
• Created: 10/Jan/19 17:35
• Updated: 12/Dec/22 18:11
• Resolved: 16/Jan/19 08:51

Description

When I try to use pandas_udf from examples in documentation:

from pyspark.sql.functions import pandas_udf, PandasUDFType

from pyspark.sql.types import IntegerType, StringType

slen = pandas_udf(lambda s: s.str.len(), IntegerType()) #here it is crashed

I get the error:

[1]    17969 illegal hardware instruction (core dumped)  python3

The environment is:

Python 3.6.7
PySpark 2.4.0
PyArrow: 0.11.1
Pandas: 0.23.4
NumPy: 1.15.4
OS: Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux