[SPARK-36093] (RemoveRedundantAliases should keep output schema name) The result incorrect if the partition path case is inconsistent

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0, 3.1.3, 3.0.4
• Component/s: SQL
• Labels: correctness
• Assignee: angerszhu
• Reporter: Yuming Wang
• Created: 12/Jul/21 14:10
• Updated: 30/Jul/21 06:57
• Resolved: 19/Jul/21 15:11

Description

Please reproduce this issue using HDFS. Local HDFS can not reproduce this issue.

sql("create table t1(cal_dt date) using parquet")
sql("insert into t1 values (date'2021-06-27'),(date'2021-06-28'),(date'2021-06-29'),(date'2021-06-30')")
sql("create view t1_v as select * from t1")
sql("CREATE TABLE t2 USING PARQUET PARTITIONED BY (CAL_DT) AS SELECT 1 AS FLAG,CAL_DT FROM t1_v WHERE CAL_DT BETWEEN '2021-06-27' AND '2021-06-28'")
sql("INSERT INTO t2 SELECT 2 AS FLAG,CAL_DT FROM t1_v WHERE CAL_DT BETWEEN '2021-06-29' AND '2021-06-30'")
sql("SELECT * FROM t2 WHERE CAL_DT BETWEEN '2021-06-29' AND '2021-06-30'").show
sql("SELECT * FROM t2 ").show

// It should not empty.
scala> sql("SELECT * FROM t2 WHERE CAL_DT BETWEEN '2021-06-29' AND '2021-06-30'").show
+----+------+
|FLAG|CAL_DT|
+----+------+
+----+------+


scala> sql("SELECT * FROM t2 ").show
+----+----------+
|FLAG|    CAL_DT|
+----+----------+
|   1|2021-06-27|
|   1|2021-06-28|
+----+----------+

scala> sql("SELECT 2 AS FLAG,CAL_DT FROM t1_v WHERE CAL_DT BETWEEN '2021-06-29' AND '2021-06-30'").show
+----+----------+
|FLAG|    CAL_DT|
+----+----------+
|   2|2021-06-29|
|   2|2021-06-30|
+----+----------+

Issue Links

• links to: [Github] Pull Request #33324 (AngersZhuuuu)
• links to: [Github] Pull Request #33417 (AngersZhuuuu)
• links to: [Github] Pull Request #33418 (AngersZhuuuu)