[SPARK-33990] v2 ALTER TABLE .. DROP PARTITION does not remove data from dropped partition

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.1.1
• Component/s: SQL
• Assignee: Max Gekk
• Reporter: Max Gekk
• Created: 04/Jan/21 12:41
• Updated: 01/Feb/21 09:20
• Resolved: 04/Jan/21 18:27

Description

The test fails:

  test("SPARK-XXXXX: don not return data from dropped partition") {
    withNamespaceAndTable("ns", "tbl") { t =>
      sql(s"CREATE TABLE $t (id int, part int) $defaultUsing PARTITIONED BY (part)")
      sql(s"INSERT INTO $t PARTITION (part=0) SELECT 0")
      sql(s"INSERT INTO $t PARTITION (part=1) SELECT 1")
      QueryTest.checkAnswer(sql(s"SELECT * FROM $t"), Seq(Row(0, 0), Row(1, 1)))
      sql(s"ALTER TABLE $t DROP PARTITION (part=0)")
      QueryTest.checkAnswer(sql(s"SELECT * FROM $t"), Seq(Row(1, 1)))
    }
  }

on the last check with:

== Results ==
!== Correct Answer - 1 ==   == Spark Answer - 2 ==
!struct<>                   struct<id:int,part:int>
![1,1]                      [0,0]
!                           [1,1]

Issue Links

• links to: [Github] Pull Request #31014 (MaxGekk)
• links to: [Github] Pull Request #31407 (yaooqinn)
• links to: [Github] Pull Request #31407 (yaooqinn)
• links to: [Github] Pull Request #31411 (MaxGekk)