Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-19359

partition path created by Hive should be deleted after rename a partition with upper-case

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • 2.2.0
    • SQL
    • None

    Description

      Hive metastore is not case preserving and keep partition columns with lower case names.

      If SparkSQL create a table with upper-case partion name use HiveExternalCatalog, when we rename partition, it first call the HiveClient to renamePartition, which will create a new lower case partition path, then SparkSql rename the lower case path to the upper-case.

      while if the renamed partition contains more than one depth partition ,e.g. A=1/B=2, hive renamePartition change to a=1/b=2, then SparkSql rename it to A=1/B=2, but the a=1 still exists in the filesystem, we should also delete it.

      Attachments

        Activity

          People

            windpiger Song Jun
            windpiger Song Jun
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: