[SPARK-38528] NullPointerException when selecting a generator in a Stream of aggregate expressions - ASF JIRA

Log work

Agile Board

Rank to Top

Rank to Bottom

Attach files

Attach Screenshot

Bulk Copy Attachments

Bulk Move Attachments

Voters

Watch issue

Watchers

Create sub-task

Convert to sub-task

Move

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

Delete

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 3.1.3, 3.2.1, 3.3.0
Fix Version/s: 3.1.3, 3.3.0, 3.2.2
Component/s: SQL
Labels:
None

Description

Assume this dataframe:

val df = Seq(1, 2, 3).toDF("v")

This works:

df.select(Seq(explode(array(min($"v"), max($"v"))), sum($"v")): _*).collect

However, this doesn't:

df.select(Stream(explode(array(min($"v"), max($"v"))), sum($"v")): _*).collect

It throws this error:

java.lang.NullPointerException
  at org.apache.spark.sql.catalyst.analysis.Analyzer$GlobalAggregates$.$anonfun$containsAggregates$1(Analyzer.scala:2516)
  at scala.collection.immutable.List.flatMap(List.scala:366)
  at org.apache.spark.sql.catalyst.analysis.Analyzer$GlobalAggregates$.containsAggregates(Analyzer.scala:2515)
  at org.apache.spark.sql.catalyst.analysis.Analyzer$GlobalAggregates$$anonfun$apply$31.applyOrElse(Analyzer.scala:2509)
  at org.apache.spark.sql.catalyst.analysis.Analyzer$GlobalAggregates$$anonfun$apply$31.applyOrElse(Analyzer.scala:2508)

The only difference between the two queries is that the first one uses Seq to specify the varargs, whereas the second one uses Stream.