[SPARK-34561] Cannot drop/add columns from/to a dataset of v2 `DESCRIBE TABLE`

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.1.2, 3.2.0
• Component/s: SQL
• Assignee: Max Gekk
• Reporter: Max Gekk
• Created: 27/Feb/21 17:06
• Updated: 01/Mar/21 14:48
• Resolved: 01/Mar/21 14:48

Description

Dropping a column from a dataset of v2 `DESCRIBE TABLE` fails with:

Resolved attribute(s) col_name#102,data_type#103 missing from col_name#29,data_type#30,comment#31 in operator !Project [col_name#102, data_type#103]. Attribute(s) with the same name appear in the operation: col_name,data_type. Please check if the right attribute(s) are used.;
!Project [col_name#102, data_type#103]
+- LocalRelation [col_name#29, data_type#30, comment#31]

The code below demonstrates the issue:

val tbl = s"${catalogAndNamespace}tbl"
withTable(tbl) {
  sql(s"CREATE TABLE $tbl (c0 INT) USING $v2Format")
  val description = sql(s"DESCRIBE TABLE $tbl")
  val noComment = description.drop("comment")
}

Issue Links

• links to: [Github] Pull Request #31676 (MaxGekk)