[SPARK-44680] parameter markers are not blocked from DEFAULT (and other places)

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.5.0, 4.0.0
• Component/s: Spark Core
• Assignee: Max Gekk
• Reporter: Serge Rielau
• Created: 04/Aug/23 16:33
• Updated: 08/Aug/23 08:26
• Resolved: 08/Aug/23 08:26

Description

scala> spark.sql("CREATE TABLE t11(c1 int default :parm)", args = Map("parm" -> 5)).show()

-> success

scala> spark.sql("describe t11");

[INVALID_DEFAULT_VALUE.UNRESOLVED_EXPRESSION] Failed to execute EXISTS_DEFAULT command because the destination table column `c1` has a DEFAULT value :parm, which fails to resolve as a valid expression.

This likely extends to other DDL-y places.
I can only find protection against placement in the body of a CREATE VIEW.

I see two ways out of this:

• Raise an error (as we do for CREATE VIEW v1(c1) AS SELECT ? )
   * Improve the way we persist queries/expressions to substitute the at-DDL-time bound parameter value (it' not a bug it's a feature....)