Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33392 Align DSv2 commands to DSv1 implementation
  3. SPARK-34543

Respect case sensitivity in V1 ALTER TABLE .. SET LOCATION

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.7, 3.0.2, 3.1.1
    • Fix Version/s: 3.0.3, 3.1.2, 3.2.0
    • Component/s: SQL
    • Labels:
      None

      Description

      SHOW PARTITIONS is case sensitive, and doesn't respect the SQL config spark.sql.caseSensitive which is false by default, for instance:

      spark-sql> CREATE TABLE tbl (id INT, part INT) PARTITIONED BY (part);
      spark-sql> INSERT INTO tbl PARTITION (part=0) SELECT 0;
      spark-sql> SHOW TABLE EXTENDED LIKE 'tbl' PARTITION (part=0);
      Location: file:/Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/part=0
      spark-sql> ALTER TABLE tbl ADD PARTITION (part=1);
      spark-sql> SELECT * FROM tbl;
      0	0
      

      Create new partition folder in the file system:

      $ cp -r /Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/part=0 /Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/aaa
      

      Set new location for the partition part=1:

      spark-sql> ALTER TABLE tbl PARTITION (part=1) SET LOCATION '/Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/aaa';
      spark-sql> SELECT * FROM tbl;
      0	0
      0	1
      spark-sql> ALTER TABLE tbl ADD PARTITION (PART=2);
      spark-sql> SELECT * FROM tbl;
      0	0
      0	1
      

      Set location for a partition in the upper case:

      $ cp -r /Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/part=0 /Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/bbb
      
      spark-sql> ALTER TABLE tbl PARTITION (PART=2) SET LOCATION '/Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/bbb';
      Error in query: Partition spec is invalid. The spec (PART) must match the partition spec (part) defined in table '`default`.`tbl`'
      

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                maxgekk Max Gekk
                Reporter:
                maxgekk Max Gekk
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: