[SPARK-46189] Various Pandas functions fail in interpreted mode

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.1, 3.5.0
• Fix Version/s: 4.0.0, 3.5.1, 3.4.3
• Component/s: Pandas API on Spark, SQL
• Labels: pull-request-available
• Assignee: Bruce Robbins
• Reporter: Bruce Robbins
• Created: 30/Nov/23 21:58
• Updated: 01/Dec/23 02:31
• Resolved: 01/Dec/23 02:29

Description

Various Pandas functions (kurt, var, skew, cov, and stddev) fail with an unboxing-related exception when run in interpreted mode.

Here are some reproduction cases for pyspark interactive mode:

spark.sql("set spark.sql.codegen.wholeStage=false")
spark.sql("set spark.sql.codegen.factoryMode=NO_CODEGEN")

import numpy as np
import pandas as pd

import pyspark.pandas as ps

pser = pd.Series([1, 2, 3, 7, 9, 8], index=np.random.rand(6), name="a")
psser = ps.from_pandas(pser)

# each of the following actions gets an unboxing error
psser.kurt()
psser.var()
psser.skew()

# set up for covariance test
pdf = pd.DataFrame([(1, 2), (0, 3), (2, 0), (1, 1)], columns=["a", "b"])
psdf = ps.from_pandas(pdf)

# this gets an unboxing error
psdf.cov()

# set up for stddev resr
from pyspark.pandas.spark import functions as SF
from pyspark.sql.functions import col
from pyspark.sql import Row
df = spark.createDataFrame([Row(a=1), Row(a=2), Row(a=3), Row(a=7), Row(a=9), Row(a=8)])

# this gets an unboxing error
df.select(SF.stddev(col("a"), 1)).collect()

Exception from the first case (psser.kurt()) is

java.lang.ClassCastException: class java.lang.Integer cannot be cast to class java.lang.Double (java.lang.Integer and java.lang.Double are in module java.base of loader 'bootstrap')
	at scala.runtime.BoxesRunTime.unboxToDouble(BoxesRunTime.java:112)
	at org.apache.spark.sql.catalyst.types.PhysicalDoubleType$$anonfun$2.compare(PhysicalDataType.scala:184)
	at scala.math.Ordering.lt(Ordering.scala:98)
	at scala.math.Ordering.lt$(Ordering.scala:98)
	at org.apache.spark.sql.catalyst.types.PhysicalDoubleType$$anonfun$2.lt(PhysicalDataType.scala:184)
	at org.apache.spark.sql.catalyst.expressions.LessThan.nullSafeEval(predicates.scala:1196)

Issue Links

• links to: GitHub Pull Request #44099