[SPARK-48995] Column.endswith(None) occasionally causes NPE

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: PySpark

Environment

Tested from pyspark shell, on Apache Spark 4.0.

• Assignee: Unassigned
• Reporter: Mithun Radhakrishnan
• Created: 24/Jul/24 21:53
• Updated: 24/Jul/24 21:55

Description

This one is pretty hard to repro, since it only seems to happen occasionally.

Invoking `Column.endswith()` seems to result in an NPE, with Spark 4.0:

from pyspark.sql.types import *
import pyspark.sql.functions as f

schema = StructType([StructField("s", StringType(), True)])

strings = [Row("abc"), Row("bcd"), Row(None)]

df = sc.parallelize(strings).toDF(schema)

df.select( f.col('s').endswith(None) ).collect()

Here is the resulting stack trace:

py4j.protocol.Py4JJavaError: An error occurred while calling o205.endsWith.
: java.lang.NullPointerException: Cannot invoke "org.apache.spark.sql.Column.expr()" because "x$1" is null
  at org.apache.spark.sql.Column$.$anonfun$fn$2(Column.scala:77)
  at scala.collection.immutable.ArraySeq.map(ArraySeq.scala:75)
  at scala.collection.immutable.ArraySeq.map(ArraySeq.scala:35)
  at org.apache.spark.sql.Column$.$anonfun$fn$1(Column.scala:77)
  at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:84)
  at org.apache.spark.sql.package$.withOrigin(package.scala:111)
  at org.apache.spark.sql.Column$.fn(Column.scala:76)
  at org.apache.spark.sql.Column$.fn(Column.scala:64)
  at org.apache.spark.sql.Column.fn(Column.scala:169)
  at org.apache.spark.sql.Column.endsWith(Column.scala:1078)
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.base/java.lang.reflect.Method.invoke(Method.java:568)
  at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
  at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
  at py4j.Gateway.invoke(Gateway.java:282)
  at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
  at py4j.commands.CallCommand.execute(CallCommand.java:79)
  at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
  at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
  at java.base/java.lang.Thread.run(Thread.java:840)

This seems to point to Column::fn, which looks new to Spark 4.0.