[SPARK-42416] Dateset operations should not resolve the analyzed logical plan again

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: SQL
• Assignee: Gengliang Wang
• Reporter: Gengliang Wang
• Created: 13/Feb/23 05:36
• Updated: 13/Feb/23 18:58
• Resolved: 13/Feb/23 18:58

Description

For the following query

 

      sql(
        """
          |CREATE TABLE app_open (
          |  uid STRING,
          |  st TIMESTAMP,
          |  ds INT
          |) USING parquet PARTITIONED BY (ds);
          |""".stripMargin)

      sql(
        """
          |create or replace temporary view group_by_error as WITH new_app_open AS (
          |  SELECT
          |    ao.*
          |  FROM
          |    app_open ao
          |)
          |SELECT
          |    uid,
          |    20230208 AS ds
          |  FROM
          |    new_app_open
          |  GROUP BY
          |    1,
          |    2
          |""".stripMargin)

      sql(
        """
          |select
          |  `uid`
          |from
          |  group_by_error
          |""".stripMargin).show()

Spark will throw the following error

 

 

[GROUP_BY_POS_OUT_OF_RANGE] GROUP BY position 20230208 is not in select list (valid range is [1, 2]).; line 9 pos 4 

 

 

This is because the logical plan is not set as analyzed and it is analyzed again. The analyzer rules about aggregation/sort ordinals are not idempotent.

Issue Links

• is caused by: SPARK-40595 Improve error message for unused CTE relations
• links to: [Github] Pull Request #39988 (gengliangwang)