Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-18455 General support for correlated subquery processing
  3. SPARK-18874

First phase: Deferring the correlated predicate pull up to Optimizer phase

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.2.0
    • Component/s: SQL
    • Labels:
      None

      Description

      This JIRA implements the first phase of SPARK-18455 by deferring the correlated predicate pull up from Analyzer to Optimizer. The goal is to preserve the current functionality of subquery in Spark 2.0 (if it works, it continues to work after this JIRA, if it does not, it won't). The performance of subquery processing is expected to be at par with Spark 2.0.

      The representation of the LogicalPlan after Analyzer will be different after this JIRA that it will preserve the original positions of correlated predicates in a subquery. This new representation is a preparation work for the second phase of extending the support of correlated subquery to cases Spark 2.0 does not support such as deep correlation, outer references in SELECT clause.

        Attachments

        1. SPARK-18874-3.pdf
          338 kB
          Nattavut Sutyanyong

          Activity

            People

            • Assignee:
              dkbiswal Dilip Biswal
              Reporter:
              nsyca Nattavut Sutyanyong
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: