[SPARK-49188] Internal error on concat_ws called on array of arrays of string

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Nikola Mandic
• Reporter: Nikola Mandic
• Created: 09/Aug/24 16:16
• Updated: 12/Aug/24 03:47
• Resolved: 12/Aug/24 03:47

Description

Applying the following sequence of queries in ANSI mode (spark.sql.ansi.enabled=true):

create table test_table(dat array<string>) using parquet;
select concat_ws(',', collect_list(dat)) FROM test_table; 

yields an internal error:

org.apache.spark.SparkException: [INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the corresponding communities or vendors, and provide the full stack trace. SQLSTATE: XX000
...
Caused by: java.lang.NullPointerException: Cannot invoke "org.apache.spark.sql.types.AbstractDataType.simpleString()" because "<local3>" is null
  at org.apache.spark.sql.errors.DataTypeErrorsBase.toSQLType(DataTypeErrorsBase.scala:55)
  at org.apache.spark.sql.errors.DataTypeErrorsBase.toSQLType$(DataTypeErrorsBase.scala:51)
  at org.apache.spark.sql.catalyst.expressions.Cast$.toSQLType(Cast.scala:44)
  at org.apache.spark.sql.catalyst.expressions.Cast$.typeCheckFailureMessage(Cast.scala:426)
  at org.apache.spark.sql.catalyst.expressions.Cast.typeCheckFailureInCast(Cast.scala:501)
  at org.apache.spark.sql.catalyst.expressions.Cast.checkInputDataTypes(Cast.scala:525)
  at org.apache.spark.sql.catalyst.expressions.Cast.resolved$lzycompute(Cast.scala:551)
  at org.apache.spark.sql.catalyst.expressions.Cast.resolved(Cast.scala:550)
... 

Issue Links

• links to: GitHub Pull Request #47691