Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33392 Align DSv2 commands to DSv1 implementation
  3. SPARK-34561

Cannot drop/add columns from/to a dataset of v2 `DESCRIBE TABLE`

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.2.0
    • Fix Version/s: 3.1.2, 3.2.0
    • Component/s: SQL
    • Labels:
      None

      Description

      Dropping a column from a dataset of v2 `DESCRIBE TABLE` fails with:

      Resolved attribute(s) col_name#102,data_type#103 missing from col_name#29,data_type#30,comment#31 in operator !Project [col_name#102, data_type#103]. Attribute(s) with the same name appear in the operation: col_name,data_type. Please check if the right attribute(s) are used.;
      !Project [col_name#102, data_type#103]
      +- LocalRelation [col_name#29, data_type#30, comment#31]

      The code below demonstrates the issue:

      val tbl = s"${catalogAndNamespace}tbl"
      withTable(tbl) {
        sql(s"CREATE TABLE $tbl (c0 INT) USING $v2Format")
        val description = sql(s"DESCRIBE TABLE $tbl")
        val noComment = description.drop("comment")
      }
      

        Attachments

          Activity

            People

            • Assignee:
              maxgekk Max Gekk
              Reporter:
              maxgekk Max Gekk
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: