Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-38236

Absolute file paths specified in create/alter table are treated as relative

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.1.1, 3.1.2, 3.2.0, 3.2.1
    • 3.1.3, 3.3.0, 3.2.2
    • SQL
    • None

    Description

      After https://github.com/apache/spark/pull/28527 we change to create table under the database location when the table location specified is relative. However the criteria to determine if a table location is relative/absolute is URI.isAbsolute, which basically checks if the table location URI has a scheme defined. So table URIs like /table/path are treated as relative and the scheme and authority of the database location URI are used to create the table. For example, when the database location URI is s3a://bucket/db, the table will be created at s3a://bucket/table/path, while it should be created under the file system defined in SessionCatalog.hadoopConf instead.

      This also applies to alter table.

      Attachments

        Activity

          People

            bozhang Bo Zhang
            bozhang Bo Zhang
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: