Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-29900

make relation lookup behavior consistent within Spark SQL

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.1.0
    • Fix Version/s: None
    • Component/s: SQL
    • Labels:
      None

      Description

      Currently, Spark has 2 different relation resolution behaviors:
      1. try to look up temp view first, then try table/persistent view.
      2. try to look up table/persistent view.

      The first behavior is used in SELECT, INSERT and a few commands that support views, like DESC TABLE.

      The second behavior is used in most commands.

      It's confusing to have inconsistent relation resolution behaviors, and the benefit is super small. It's only useful when there are temp view and table with the same name, but users can easily use qualified table name to disambiguate.

      In postgres, the relation resolution behavior is consistent

      cloud0fan=# create schema s1;
      CREATE SCHEMA
      cloud0fan=# SET search_path TO s1;
      SET
      cloud0fan=# create table s1.t (i int);
      CREATE TABLE
      cloud0fan=# insert into s1.t values (1);
      INSERT 0 1
      
      # access table with qualified name
      cloud0fan=# select * from s1.t;
       i 
      ---
       1
      (1 row)
      
      # access table with single name
      cloud0fan=# select * from t;
       i 
      ---
       1
      (1 rows)
      
      # create a temp view with conflicting name
      cloud0fan=# create temp view t as select 2 as i;
      CREATE VIEW
      
      # same as spark, temp view has higher proirity during resolution
      cloud0fan=# select * from t;
       i 
      ---
       2
      (1 row)
      
      # DROP TABLE also resolves temp view first
      cloud0fan=# drop table t;
      ERROR:  "t" is not a table
      
      # DELETE also resolves temp view first
      cloud0fan=# delete from t where i = 0;
      ERROR:  cannot delete from view "t"
      

        Attachments

          Issue Links

          1.
          Migrate SHOW TBLPROPERTIES to new framework Sub-task Resolved Terry Kim
          2.
          V2 Command logical plan should use UnresolvedV2Relation for a table Sub-task Resolved Unassigned
          3.
          Commands involved with namespace go thru the new resolution framework. Sub-task Resolved Terry Kim
          4.
          Migrate ALTER TABLE commands to the new resolution framework Sub-task Resolved Unassigned
          5.
          Migrate ALTER VIEW SET/UNSET commands to the new resolution framework Sub-task Resolved Unassigned
          6.
          Introduce UnresolvedTableOrPermanentView for commands that support a table/view but not a temporary view Sub-task Resolved Unassigned
          7.
          Migrate function related commands to new framework Sub-task Resolved Terry Kim
          8.
          Migrate REFRESH TABLE to new resolution framework Sub-task Resolved Terry Kim
          9.
          Migrate DESCRIBE COLUMN to new resolution framework Sub-task Resolved Terry Kim
          10.
          Migrate DROP TABLE to new resolution framework Sub-task Resolved Terry Kim
          11.
          Migrate ANALYZE TABLE to new resolution framework Sub-task Resolved Terry Kim
          12.
          Migrate LOAD DATA to new resolution framework Sub-task Resolved Terry Kim
          13.
          Migrate SHOW CREATE TABLE to new resolution framework Sub-task Resolved Terry Kim
          14.
          Support CACHE/UNCACHE TABLE for v2 tables Sub-task Resolved Terry Kim
          15.
          Migrate TRUNCATE TABLE to new resolution framework Sub-task Resolved Terry Kim
          16.
          Improve exception messages while handling UnresolvedTable Sub-task Resolved Terry Kim
          17.
          Improve exception messages while handling UnresolvedTableOrView Sub-task Resolved Terry Kim
          18.
          Migrate SHOW COLUMNS to new resolution framework Sub-task Resolved Terry Kim
          19.
          Migrate CACHE TABLE to new resolution framework Sub-task Resolved Terry Kim
          20.
          Migrate ALTER TABLE ... RENAME TO to new resolution framework Sub-task Resolved Terry Kim
          21.
          Migrate DROP VIEW to the new resolution framework Sub-task Resolved Terry Kim
          22.
          Migrate SHOW TABLE EXTENDED to new resolution framework Sub-task Resolved Max Gekk
          23.
          Migrate MSCK REPAIR TABLE to the new resolution framework Sub-task Resolved Terry Kim
          24.
          Migrate ALTER VIEW ... SET/UNSET TBLPROPERTIES to new resolution framework Sub-task Resolved Terry Kim
          25.
          Migrate ALTER VIEW ... AS command to the new resolution framework Sub-task Resolved Terry Kim
          26.
          Migrate UNCACHE TABLE to new resolution framework Sub-task Resolved Terry Kim
          27.
          Migrate ALTER TABLE ... RECOVER PARTITIONS to new resolution framework Sub-task Resolved Terry Kim
          28.
          Migrate ALTER TABLE SET [SERDE|SERDEPROPERTIES] command to the new resolution framework Sub-task Resolved Terry Kim
          29.
          Migrate ALTER TABLE ... RENAME TO PARTITION commands to the new resolution framework Sub-task Resolved Terry Kim
          30.
          Migrate ALTER TABLE .. CHANGE COLUMN to new resolution framework Sub-task Resolved Terry Kim
          31.
          Migrate ALTER TABLE ... SET LOCATION to new resolution framework Sub-task Resolved Max Gekk
          32.
          Migrate ALTER TABLE SET/UNSET TBLPROPERTIES commands to the new resolution framework Sub-task Resolved Terry Kim
          33.
          Migrate ALTER TABLE drop columns command to the new resolution framework Sub-task Resolved Terry Kim
          34.
          Migrate ALTER TABLE commands that alter columns to new framework Sub-task Resolved Unassigned
          35.
          Migrate ALTER TABLE rename column command to the new resolution framework Sub-task Resolved Terry Kim
          36.
          Migrate ALTER TABLE ADD/RENAME COLUMNS command to the new resolution framework Sub-task Resolved Terry Kim
          37.
          Simplify the logical plan names for ALTER TABLE ... COLUMN Sub-task Resolved Wenchen Fan

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                cloud_fan Wenchen Fan
              • Votes:
                0 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated: