Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25603 Generalize Nested Column Pruning
  3. SPARK-26402

Accessing nested fields with different cases in case insensitive mode

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.4.1, 3.0.0
    • Component/s: SQL
    • Labels:
      None
    • Target Version/s:

      Description

      GetStructField with different optional names should be semantically equal. We will use this as building block to compare the nested fields used in the plans to be optimized by catalyst optimizer.

      This PR also fixes a bug below that accessing nested fields with different cases in case insensitive mode will result result AnalysisException.

      sql("create table t (s struct<i: Int>) using json")
      sql("select s.I from t group by s.i")
      

      which is currently failing

      org.apache.spark.sql.AnalysisException: expression 'default.t.`s`' is neither present in the group by, nor is it an aggregate function
      

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                dbtsai DB Tsai
                Reporter:
                dbtsai DB Tsai
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: