[SPARK-34543] Respect case sensitivity in V1 ALTER TABLE .. SET LOCATION

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.7, 3.0.2, 3.1.1
• Fix Version/s: 3.0.3, 3.1.2, 3.2.0
• Component/s: SQL
• Assignee: Max Gekk
• Reporter: Max Gekk
• Created: 25/Feb/21 15:18
• Updated: 28/Feb/21 05:04
• Resolved: 26/Feb/21 07:30

Description

SHOW PARTITIONS is case sensitive, and doesn't respect the SQL config spark.sql.caseSensitive which is false by default, for instance:

spark-sql> CREATE TABLE tbl (id INT, part INT) PARTITIONED BY (part);
spark-sql> INSERT INTO tbl PARTITION (part=0) SELECT 0;
spark-sql> SHOW TABLE EXTENDED LIKE 'tbl' PARTITION (part=0);
Location: file:/Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/part=0
spark-sql> ALTER TABLE tbl ADD PARTITION (part=1);
spark-sql> SELECT * FROM tbl;
0	0

Create new partition folder in the file system:

$ cp -r /Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/part=0 /Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/aaa

Set new location for the partition part=1:

spark-sql> ALTER TABLE tbl PARTITION (part=1) SET LOCATION '/Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/aaa';
spark-sql> SELECT * FROM tbl;
0	0
0	1
spark-sql> ALTER TABLE tbl ADD PARTITION (PART=2);
spark-sql> SELECT * FROM tbl;
0	0
0	1

Set location for a partition in the upper case:

$ cp -r /Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/part=0 /Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/bbb

spark-sql> ALTER TABLE tbl PARTITION (PART=2) SET LOCATION '/Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/bbb';
Error in query: Partition spec is invalid. The spec (PART) must match the partition spec (part) defined in table '`default`.`tbl`'

 

Issue Links

• is a clone of: SPARK-33667 Respect case sensitivity in V1 SHOW PARTITIONS
• links to: [Github] Pull Request #31651 (MaxGekk)
• links to: [Github] Pull Request #31651 (MaxGekk)
• links to: [Github] Pull Request #31672 (MaxGekk)