Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33392 Align DSv2 commands to DSv1 implementation
  3. SPARK-34543

Respect case sensitivity in V1 ALTER TABLE .. SET LOCATION

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.4.7, 3.0.2, 3.1.1
    • 3.0.3, 3.1.2, 3.2.0
    • SQL
    • None

    Description

      SHOW PARTITIONS is case sensitive, and doesn't respect the SQL config spark.sql.caseSensitive which is false by default, for instance:

      spark-sql> CREATE TABLE tbl (id INT, part INT) PARTITIONED BY (part);
      spark-sql> INSERT INTO tbl PARTITION (part=0) SELECT 0;
      spark-sql> SHOW TABLE EXTENDED LIKE 'tbl' PARTITION (part=0);
      Location: file:/Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/part=0
      spark-sql> ALTER TABLE tbl ADD PARTITION (part=1);
      spark-sql> SELECT * FROM tbl;
      0	0
      

      Create new partition folder in the file system:

      $ cp -r /Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/part=0 /Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/aaa
      

      Set new location for the partition part=1:

      spark-sql> ALTER TABLE tbl PARTITION (part=1) SET LOCATION '/Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/aaa';
      spark-sql> SELECT * FROM tbl;
      0	0
      0	1
      spark-sql> ALTER TABLE tbl ADD PARTITION (PART=2);
      spark-sql> SELECT * FROM tbl;
      0	0
      0	1
      

      Set location for a partition in the upper case:

      $ cp -r /Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/part=0 /Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/bbb
      
      spark-sql> ALTER TABLE tbl PARTITION (PART=2) SET LOCATION '/Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/bbb';
      Error in query: Partition spec is invalid. The spec (PART) must match the partition spec (part) defined in table '`default`.`tbl`'
      

       

      Attachments

        Issue Links

          Activity

            People

              maxgekk Max Gekk
              maxgekk Max Gekk
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: