Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33392

Align DSv2 commands to DSv1 implementation

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.1.0
    • None
    • SQL
    • None

    Description

      The purpose of this umbrella ticket is:

      1. Implement missing features of datasource v1 commands in DSv2
      2. Align behavior of DSv2 commands to the current implementation of DSv1 commands as much as possible.

      Attachments

        Issue Links

        is related to

        Improvement - An improvement or enhancement to an existing feature or task. SPARK-33507 Improve and fix cache behavior in v1 and v2

        • Critical - Crashes, loss of data, severe memory leak.
        • Open
        relates to

        Test - A new unit, integration or system test. SPARK-33381 Unify DSv1 and DSv2 command tests

        • Major - Major loss of function.
        • Open
        1.
        		 Expose purge option in TableCatalog.dropTable Sub-task Resolved Terry Kim Actions
        2.
        		 Support SHOW TABLE EXTENDED in DSv2 Sub-task Resolved Pan Bingkun Actions
        3.
        		 Throw `NoSuchDatabaseException` for not existing namespace in DSv2 SHOW TABLES Sub-task Resolved Max Gekk Actions
        4.
        		 DSv2 SHOW TABLES doesn't show `default` Sub-task Open Unassigned Actions
        5.
        		 Create a V2 SHOW PARTITIONS execution node Sub-task Resolved Max Gekk Actions
        6.
        		 List partition by names from V2 tables that support partition management Sub-task Resolved Max Gekk Actions
        7.
        		 Respect case sensitivity in resolving partition specs V2 Sub-task Resolved Max Gekk Actions
        8.
        		 Universal type conversion of V2 partition values Sub-task Resolved Max Gekk Actions
        9.
        		 Resolver of V2 partition specs doesn't handle __HIVE_DEFAULT_PARTITION__ Sub-task In Progress Apache Spark Actions
        10.
        		 Remove getting partitions by only ident Sub-task Resolved Max Gekk Actions
        11.
        		 Misleading error from ALTER TABLE .. PARTITION for non-supported partition management table Sub-task Resolved Max Gekk Actions
        12.
        		 Respect case sensitivity in V1 SHOW PARTITIONS Sub-task Resolved Max Gekk Actions
        13.
        		 Verify the partition provider is Hive in v1 SHOW TABLE EXTENDED Sub-task Resolved Max Gekk Actions
        14.
        		 Remove VIEW checks from V1 table commands Sub-task Resolved Max Gekk Actions
        15.
        		 Require exact matched partition spec to schema in ADD/DROP PARTITION Sub-task Resolved Max Gekk Actions
        16.
        		 Require fully specified partition identifier in partitionExists() Sub-task Resolved Max Gekk Actions
        17.
        		 Throw PartitionsAlreadyExistException from HiveExternalCatalog.createPartitions() Sub-task Resolved Max Gekk Actions
        18.
        		 Remove unused parameter `retainData` from AlterTableDropPartition Sub-task Resolved Max Gekk Actions
        19.
        		 Sort output of V2 SHOW PARTITIONS Sub-task Resolved Max Gekk Actions
        20.
        		 Support partition purging by `SupportsPartitionManagement` Sub-task Resolved Max Gekk Actions
        21.
        		 Support V2 ALTER TABLE .. RENAME PARTITION Sub-task Resolved Max Gekk Actions
        22.
        		 Throw NoSuchPartitionsException from HiveExternalCatalog.dropPartitions() Sub-task Resolved Max Gekk Actions
        23.
        		 Describe the PURGE option of ALTER TABLE .. DROP PARTITION Sub-task Resolved Unassigned Actions
        24.
        		 Add comments about `PURGE` in DropTable and in AlterTableDropPartition Sub-task Resolved Max Gekk Actions
        25.
        		 Throw `PartitionAlreadyExistsException` if the target partition exists in Hive v1 Table Catalog Sub-task Resolved Max Gekk Actions
        26.
        		 Implement DESCRIBE COLUMN for v2 catalog Sub-task Resolved Terry Kim Actions
        27.
        		 NPE from SHOW PARTITIONS for null partition values Sub-task Resolved Max Gekk Actions
        28.
        		 v1 SHOW TABLES fails with assert on spark_catalog Sub-task Resolved Max Gekk Actions
        29.
        		 CREATE TABLE LIKE FOR V2 Sub-task Open Unassigned Actions
        30.
        		 Support SHOW CREATE TABLE in v2 Sub-task Resolved PengLei Actions
        31.
        		 CREATE VIEW AS SELECT FOR V2 Sub-task Open Unassigned Actions
        32.
        		 Recognize `spark_catalog` in `saveAsTable()` and `insertInto()` Sub-task Resolved Max Gekk Actions
        33.
        		 Update SQL migration guide about changes in HiveClientImpl Sub-task Resolved Max Gekk Actions
        34.
        		 Improve the error message in resolving of DSv1 multi-part identifiers Sub-task Resolved Max Gekk Actions
        35.
        		 CACHE TABLE does not support `spark_catalog` in Hive table names Sub-task Resolved Max Gekk Actions
        36.
        		 v2 ALTER TABLE .. DROP PARTITION does not refresh cached table Sub-task Resolved Max Gekk Actions
        37.
        		 v2 ALTER TABLE .. DROP PARTITION does not remove data from dropped partition Sub-task Resolved Max Gekk Actions
        38.
        		 Support v2 TRUNCATE TABLE Sub-task Resolved Max Gekk Actions
        39.
        		 Use logical plan of alter table in `CatalogImpl.recoverPartitions()` Sub-task Resolved Max Gekk Actions
        40.
        		 Remove view checks in v1 alter table commands Sub-task Resolved Apache Spark Actions
        41.
        		 Support partition truncation by `SupportsPartitionManagement` Sub-task Resolved Max Gekk Actions
        42.
        		 Support table truncation by v2 Table Catalogs Sub-task Resolved Max Gekk Actions
        43.
        		 Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES Sub-task In Progress Terry Kim Actions
        44.
        		 Support v2 `MSCK REPAIR TABLE` Sub-task Open Unassigned Actions
        45.
        		 Recognize `spark_catalog` in new identifier while view/table renaming Sub-task Resolved Unassigned Actions
        46.
        		 Rename alter table exec nodes Sub-task Resolved Max Gekk Actions
        47.
        		 Fix v2 ALTER TABLE .. RENAME TO Sub-task Resolved Max Gekk Actions
        48.
        		 Rename v2 logical nodes Sub-task Resolved Max Gekk Actions
        49.
        		 Respect case sensitivity in V1 ALTER TABLE .. SET LOCATION Sub-task Resolved Max Gekk Actions
        50.
        		 Cannot drop/add columns from/to a dataset of v2 `DESCRIBE TABLE` Sub-task Resolved Max Gekk Actions
        51.
        		 Cannot drop/add columns from/to a dataset of v2 `DESCRIBE COLUMN` Sub-task Resolved Unassigned Actions
        52.
        		 Cannot drop/add columns from/to a dataset of v2 `DESCRIBE NAMESPACE` Sub-task Resolved angerszhu Actions
        53.
        		 Unify v1 and v2 DESCRIBE TABLE Sub-task Resolved Max Gekk Actions
        54.
        		 Support v2 SHOW FUNCTIONS Sub-task Resolved Max Gekk Actions
        55.
        		 Support v2 DESCRIBE TABLE .. PARTITION Sub-task In Progress Unassigned Actions
        56.
        		 Support v2 `DESCRIBE TABLE EXTENDED` for columns Sub-task Resolved Huaxin Gao Actions
        57.
        		 Support v2 SHOW COLLATIONS Sub-task Open Unassigned Actions

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            maxgekk Max Gekk
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:

              Slack

                Issue deployment