Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-40804

Missing handling a catalog name in destination tables in `RenameTableExec`

    XMLWordPrintableJSON

Details

    • Bug
    • Status: In Progress
    • Major
    • Resolution: Unresolved
    • 3.1.3, 3.3.0, 3.2.2
    • 3.1.3, 3.3.0, 3.2.2
    • SQL
    • None
    • This depends on Spark versions, since the version in which ALTER TABLE RENAME TO is added.

    Description

      Currently `RenameTableExec` only handles an empty namespace for destination tables as the following current spec:

      (3.3.0 is picked up) https://github.com/apache/spark/blob/v3.3.0/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/RenameTableExec.scala#L48

       

      # L48
          val qualifiedNewIdent = if (newIdent.namespace.isEmpty) {
            Identifier.of(oldIdent.namespace, newIdent.name)
          } else newIdent

      This part doesn't handle the case where a destination table is specified catalog.db.table. For example, Apache Iceberg is expected to handle the destination table as catalog.db.table in the document; https://iceberg.apache.org/docs/latest/spark-ddl/#alter-table--rename-to

       

      If catalog.db.table is passed to "ALTER TABLE <src> RENAME TO <dst>" query, there's a difference of handling namespaces between source and destination tables. Specifically, source tables can be correctly handled as [db] for its namespace, but destination tables are handled as [catalog, db] for its namespace.

      Attachments

        Activity

          People

            Unassigned Unassigned
            tom_tanaka Tomohiro Tanaka
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - 48h
                48h
                Remaining:
                Remaining Estimate - 48h
                48h
                Logged:
                Time Spent - Not Specified
                Not Specified