Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33392 Align DSv2 commands to DSv1 implementation
  3. SPARK-34561

Cannot drop/add columns from/to a dataset of v2 `DESCRIBE TABLE`

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersStop watchingWatchersConvert to IssueLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.0
    • 3.1.2, 3.2.0
    • SQL
    • None

    Description

      Dropping a column from a dataset of v2 `DESCRIBE TABLE` fails with:

      Resolved attribute(s) col_name#102,data_type#103 missing from col_name#29,data_type#30,comment#31 in operator !Project [col_name#102, data_type#103]. Attribute(s) with the same name appear in the operation: col_name,data_type. Please check if the right attribute(s) are used.;
      !Project [col_name#102, data_type#103]
      +- LocalRelation [col_name#29, data_type#30, comment#31]

      The code below demonstrates the issue:

      val tbl = s"${catalogAndNamespace}tbl"
      withTable(tbl) {
        sql(s"CREATE TABLE $tbl (c0 INT) USING $v2Format")
        val description = sql(s"DESCRIBE TABLE $tbl")
        val noComment = description.drop("comment")
      }
      

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            maxgekk Max Gekk Assign to me
            maxgekk Max Gekk
            Votes:
            0 Vote for this issue
            Watchers:
            2 Stop watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment