Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33392

Align DSv2 commands to DSv1 implementation

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.1.0
    • None
    • SQL
    • None

    Description

      The purpose of this umbrella ticket is:

      1. Implement missing features of datasource v1 commands in DSv2
      2. Align behavior of DSv2 commands to the current implementation of DSv1 commands as much as possible.

      Attachments

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-33507 Improve and fix cache behavior in v1 and v2

          • Critical - Crashes, loss of data, severe memory leak.
          • Open
          relates to

          Test - A new unit, integration or system test. SPARK-33381 Unify DSv1 and DSv2 command tests

          • Major - Major loss of function.
          • Open
          1.
          		 Expose purge option in TableCatalog.dropTable Sub-task Resolved Terry Kim
          2.
          		 Support SHOW TABLE EXTENDED in DSv2 Sub-task Resolved Pan Bingkun
          3.
          		 Throw `NoSuchDatabaseException` for not existing namespace in DSv2 SHOW TABLES Sub-task Resolved Max Gekk
          4.
          		 DSv2 SHOW TABLES doesn't show `default` Sub-task Open Unassigned
          5.
          		 Create a V2 SHOW PARTITIONS execution node Sub-task Resolved Max Gekk
          6.
          		 List partition by names from V2 tables that support partition management Sub-task Resolved Max Gekk
          7.
          		 Respect case sensitivity in resolving partition specs V2 Sub-task Resolved Max Gekk
          8.
          		 Universal type conversion of V2 partition values Sub-task Resolved Max Gekk
          9.
          		 Resolver of V2 partition specs doesn't handle __HIVE_DEFAULT_PARTITION__ Sub-task In Progress Apache Spark
          10.
          		 Remove getting partitions by only ident Sub-task Resolved Max Gekk
          11.
          		 Misleading error from ALTER TABLE .. PARTITION for non-supported partition management table Sub-task Resolved Max Gekk
          12.
          		 Respect case sensitivity in V1 SHOW PARTITIONS Sub-task Resolved Max Gekk
          13.
          		 Verify the partition provider is Hive in v1 SHOW TABLE EXTENDED Sub-task Resolved Max Gekk
          14.
          		 Remove VIEW checks from V1 table commands Sub-task Resolved Max Gekk
          15.
          		 Require exact matched partition spec to schema in ADD/DROP PARTITION Sub-task Resolved Max Gekk
          16.
          		 Require fully specified partition identifier in partitionExists() Sub-task Resolved Max Gekk
          17.
          		 Throw PartitionsAlreadyExistException from HiveExternalCatalog.createPartitions() Sub-task Resolved Max Gekk
          18.
          		 Remove unused parameter `retainData` from AlterTableDropPartition Sub-task Resolved Max Gekk
          19.
          		 Sort output of V2 SHOW PARTITIONS Sub-task Resolved Max Gekk
          20.
          		 Support partition purging by `SupportsPartitionManagement` Sub-task Resolved Max Gekk
          21.
          		 Support V2 ALTER TABLE .. RENAME PARTITION Sub-task Resolved Max Gekk
          22.
          		 Throw NoSuchPartitionsException from HiveExternalCatalog.dropPartitions() Sub-task Resolved Max Gekk
          23.
          		 Describe the PURGE option of ALTER TABLE .. DROP PARTITION Sub-task Resolved Unassigned
          24.
          		 Add comments about `PURGE` in DropTable and in AlterTableDropPartition Sub-task Resolved Max Gekk
          25.
          		 Throw `PartitionAlreadyExistsException` if the target partition exists in Hive v1 Table Catalog Sub-task Resolved Max Gekk
          26.
          		 Implement DESCRIBE COLUMN for v2 catalog Sub-task Resolved Terry Kim
          27.
          		 NPE from SHOW PARTITIONS for null partition values Sub-task Resolved Max Gekk
          28.
          		 v1 SHOW TABLES fails with assert on spark_catalog Sub-task Resolved Max Gekk
          29.
          		 CREATE TABLE LIKE FOR V2 Sub-task Open Unassigned
          30.
          		 Support SHOW CREATE TABLE in v2 Sub-task Resolved PengLei
          31.
          		 CREATE VIEW AS SELECT FOR V2 Sub-task Open Unassigned
          32.
          		 Recognize `spark_catalog` in `saveAsTable()` and `insertInto()` Sub-task Resolved Max Gekk
          33.
          		 Update SQL migration guide about changes in HiveClientImpl Sub-task Resolved Max Gekk
          34.
          		 Improve the error message in resolving of DSv1 multi-part identifiers Sub-task Resolved Max Gekk
          35.
          		 CACHE TABLE does not support `spark_catalog` in Hive table names Sub-task Resolved Max Gekk
          36.
          		 v2 ALTER TABLE .. DROP PARTITION does not refresh cached table Sub-task Resolved Max Gekk
          37.
          		 v2 ALTER TABLE .. DROP PARTITION does not remove data from dropped partition Sub-task Resolved Max Gekk
          38.
          		 Support v2 TRUNCATE TABLE Sub-task Resolved Max Gekk
          39.
          		 Use logical plan of alter table in `CatalogImpl.recoverPartitions()` Sub-task Resolved Max Gekk
          40.
          		 Remove view checks in v1 alter table commands Sub-task Resolved Apache Spark
          41.
          		 Support partition truncation by `SupportsPartitionManagement` Sub-task Resolved Max Gekk
          42.
          		 Support table truncation by v2 Table Catalogs Sub-task Resolved Max Gekk
          43.
          		 Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES Sub-task In Progress Terry Kim
          44.
          		 Support v2 `MSCK REPAIR TABLE` Sub-task Open Unassigned
          45.
          		 Recognize `spark_catalog` in new identifier while view/table renaming Sub-task Resolved Unassigned
          46.
          		 Rename alter table exec nodes Sub-task Resolved Max Gekk
          47.
          		 Fix v2 ALTER TABLE .. RENAME TO Sub-task Resolved Max Gekk
          48.
          		 Rename v2 logical nodes Sub-task Resolved Max Gekk
          49.
          		 Respect case sensitivity in V1 ALTER TABLE .. SET LOCATION Sub-task Resolved Max Gekk
          50.
          		 Cannot drop/add columns from/to a dataset of v2 `DESCRIBE TABLE` Sub-task Resolved Max Gekk
          51.
          		 Cannot drop/add columns from/to a dataset of v2 `DESCRIBE COLUMN` Sub-task Resolved Unassigned
          52.
          		 Cannot drop/add columns from/to a dataset of v2 `DESCRIBE NAMESPACE` Sub-task Resolved angerszhu
          53.
          		 Unify v1 and v2 DESCRIBE TABLE Sub-task Resolved Max Gekk
          54.
          		 Support v2 SHOW FUNCTIONS Sub-task Resolved Max Gekk
          55.
          		 Support v2 DESCRIBE TABLE .. PARTITION Sub-task In Progress Unassigned
          56.
          		 Support v2 `DESCRIBE TABLE EXTENDED` for columns Sub-task Resolved Huaxin Gao
          57.
          		 Support v2 SHOW COLLATIONS Sub-task Open Unassigned

          Activity

            People

              Unassigned Unassigned
              maxgekk Max Gekk
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated: