[SPARK-40535] NPE from observe of collect_list

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.3.1, 3.4.0
• Component/s: SQL
• Assignee: Jiaan Geng
• Reporter: Max Gekk
• Created: 22/Sep/22 13:03
• Updated: 16/Oct/22 08:52
• Resolved: 23/Sep/22 11:24

Description

The code below reproduces the issue:

import org.apache.spark.sql.functions._
val df = spark.range(1,10,1,11)
df.observe("collectedList", collect_list("id")).collect()

instead of

Array(1, 2, 3, 4, 5, 6, 7, 8, 9)

it fails with the NPE:

java.lang.NullPointerException
	at org.apache.spark.sql.catalyst.expressions.aggregate.TypedImperativeAggregate.getBufferObject(interfaces.scala:641)
	at org.apache.spark.sql.catalyst.expressions.aggregate.TypedImperativeAggregate.getBufferObject(interfaces.scala:602)
	at org.apache.spark.sql.catalyst.expressions.aggregate.TypedImperativeAggregate.serializeAggregateBufferInPlace(interfaces.scala:624)
	at org.apache.spark.sql.execution.AggregatingAccumulator.withBufferSerialized(AggregatingAccumulator.scala:205)
	at org.apache.spark.sql.execution.AggregatingAccumulator.withBufferSerialized(AggregatingAccumulator.scala:33)

Issue Links

• links to: [Github] Pull Request #37977 (beliefer)
• links to: [Github] Pull Request #37977 (beliefer)