[SPARK-34055] ALTER TABLE .. ADD PARTITION doesn't refresh cache - ASF JIRA

Details

Type: Sub-task
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 3.0.1, 3.1.0, 3.2.0
Fix Version/s: 3.0.2, 3.1.1, 3.2.0
Component/s: SQL
Labels:
- correctness

Description

Here is the example to reproduce the issue:

spark-sql> create table tbl2 (col int, part int) partitioned by (part);
spark-sql> insert into tbl2 partition (part=0) select 0;
spark-sql> cache table tbl2;
spark-sql> select * from tbl2;
0	0
spark-sql> show table extended like 'tbl2' partition (part = 0);
default	tbl2	false	Partition Values: [part=0]
Location: file:/Users/maximgekk/proj/add-partition-refresh-cache-2/spark-warehouse/tbl2/part=0
...

Add new partition by copying the existing one:

cp -r /Users/maximgekk/proj/add-partition-refresh-cache-2/spark-warehouse/tbl2/part=0 /Users/maximgekk/proj/add-partition-refresh-cache-2/spark-warehouse/tbl2/part=1

Add new partition and select the table:

spark-sql> alter table tbl2 add partition (part = 1) location '/Users/maximgekk/proj/add-partition-refresh-cache-2/spark-warehouse/tbl2/part=1';
spark-sql> select * from tbl2;
0	0

We see only old data.