[SPARK-26402] Accessing nested fields with different cases in case insensitive mode

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.1, 3.0.0
• Component/s: SQL
• Target Version/s: 3.0.0
• Assignee: DB Tsai
• Reporter: DB Tsai
• Created: 19/Dec/18 00:06
• Updated: 17/Mar/20 03:55
• Resolved: 05/Jan/19 23:48

Description

GetStructField with different optional names should be semantically equal. We will use this as building block to compare the nested fields used in the plans to be optimized by catalyst optimizer.

This PR also fixes a bug below that accessing nested fields with different cases in case insensitive mode will result result AnalysisException.

sql("create table t (s struct<i: Int>) using json")
sql("select s.I from t group by s.i")

which is currently failing

org.apache.spark.sql.AnalysisException: expression 'default.t.`s`' is neither present in the group by, nor is it an aggregate function

 

Issue Links

• is related to: SPARK-31116 PrquetRowConverter does not follow case sensitivity
• links to: GitHub Pull Request #23353