Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-41154

Incorrect relation caching for queries with time travel spec

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.0, 3.3.1
    • 3.3.2, 3.4.0
    • SQL
    • None

    Description

      https://github.com/apache/spark/pull/34497 added AS OF syntax support to support time travel queries in SQL. When resolving these we cache the resolved relation with only the qualified table name as the key, ignoring the time travel spec. Thus any subsequent queries on that table are resolved using the first's time travel spec.

      This affects subqueries, CTEs, and temporary views (when created with SQL).

      Queries like this will be incorrectly resolved:

      select * from table version as of 1
      union all
      select * from table version as of 0
      

      --->

      select * from table version as of 1
      union all
      select * from table version as of 1
      

      This was originally reported here https://github.com/delta-io/delta/issues/1479

      Attachments

        Activity

          People

            ulysses XiDuo You
            allison-portis Allison Portis
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: