Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-35553 Improve correlated subqueries
  3. SPARK-40615

Check unsupported data type when decorrelating subqueries

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.4.0
    • 3.4.0
    • SQL
    • None

    Description

      Currently, we don't check if the join condition after subquery decorrelation is resolved or not. This can lead to confusing error messages to users. We should throw proper exceptions when one or more join conditions after decorrelation are not resolved.

      For example:

      create temp view v1(x) as values from_json('{"a":1, "b":2}', 'map<string,int>') t(x)
      
      select (select a + a from (select upper(x['a']) as a)) from v1 
      
      After applying rule org.apache.spark.sql.catalyst.optimizer.PullupCorrelatedPredicates in batch Pullup Correlated Expressions, the structural integrity of the plan is broken.

       

      Attachments

        Activity

          People

            allisonwang-db Allison Wang
            allisonwang-db Allison Wang
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: