Details
Blocker Description
The accumulator of ArrayAggregate should copy the intermediate result if string, struct, array, or map.
import org.apache.spark.sql.functions._ val reverse = udf((s: String) => s.reverse) val df = Seq(Array("abc", "def")).toDF("array") val testArray = df.withColumn( "agg", aggregate( col("array"), array().cast("array<string>"), (acc, s) => concat(acc, array(reverse(s))))) aggArray.show(truncate=false)
should be:
+----------+----------+ |array |agg | +----------+----------+ |[abc, def]|[cba, fed]| +----------+----------+
but:
+----------+----------+ |array |agg | +----------+----------+ |[abc, def]|[fed, fed]| +----------+----------+
