[SPARK-44313] Generated column expression validation fails if there is a char/varchar column anywhere in the schema

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0, 3.4.1
• Fix Version/s: 3.4.2, 3.5.0
• Component/s: SQL
• Assignee: Allison Portis
• Reporter: Allison Portis
• Created: 05/Jul/23 22:37
• Updated: 06/Jul/23 02:27
• Resolved: 06/Jul/23 02:27

Description

When validating generated column expressions, this call https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/GeneratedColumn.scala#L123 to checkAnalysis fails when there are char or varchar columns anywhere in the schema.

 

For example, this query will fail

CREATE TABLE default.example (
    name VARCHAR(64),
    tstamp TIMESTAMP,
    tstamp_date DATE GENERATED ALWAYS AS (CAST(tstamp as DATE))
)

 

Issue Links

• relates to: SPARK-41290 Support GENERATED ALWAYS AS syntax in create/replace table to create a generated column