[SPARK-38097] Improve the error for pivoting of unsupported value types

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: SQL
• Assignee: Yuto Akutsu
• Reporter: Max Gekk
• Created: 03/Feb/22 10:31
• Updated: 14/Feb/22 12:25
• Resolved: 14/Feb/22 12:25

Description

The error message from:

  test("Improve the error for pivoting of unsupported value types") {
    trainingSales
      .groupBy($"sales.year")
      .pivot(struct(lower($"sales.course"), $"training"))
      .agg(sum($"sales.earnings"))
      .show(false)
  }

can confuse users:

The feature is not supported: literal for '[dotnet,Dummies]' of class org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema.
org.apache.spark.SparkRuntimeException: The feature is not supported: literal for '[dotnet,Dummies]' of class org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema.
	at org.apache.spark.sql.errors.QueryExecutionErrors$.literalTypeUnsupportedError(QueryExecutionErrors.scala:245)
	at org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:99)
	at org.apache.spark.sql.RelationalGroupedDataset.$anonfun$pivot$2(RelationalGroupedDataset.scala:455)
	at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)

Need to improve the error message and make it more precise.

See https://github.com/apache/spark/pull/35302#discussion_r793629370

Issue Links

• links to: [Github] Pull Request #35433 (yutoacts)