Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24669

Managed table was not cleared of path after drop database cascade

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.3.0, 2.3.1
    • 2.3.4, 2.4.1, 3.0.0
    • SQL
    • None

    Description

      I can do the following in sequence

      1. Create a managed table using path options
      2. Drop the table via dropping the parent database cascade
      3. Re-create the database and table with a different path
      4. The new table shows data from the old path, not the new path
      echo "first" > /tmp/first.csv
      echo "second" > /tmp/second.csv
      spark-shell
      spark.version
      res0: String = 2.3.0
      spark.sql("create database foo")
      spark.sql("create table foo.first (id string) using csv options (path='/tmp/first.csv')")
      spark.table("foo.first").show()
      +-----+
      |   id|
      +-----+
      |first|
      +-----+
      spark.sql("drop database foo cascade")
      spark.sql("create database foo")
      spark.sql("create table foo.first (id string) using csv options (path='/tmp/second.csv')")
      "note, the path is different now, pointing to second.csv, but still showing data from first file"
      spark.table("foo.first").show()
      +-----+
      |   id|
      +-----+
      |first|
      +-----+
      "now, if I drop the table explicitly, instead of via dropping database cascade, then it will be the correct result"
      spark.sql("drop table foo.first")
      spark.sql("create table foo.first (id string) using csv options (path='/tmp/second.csv')")
      spark.table("foo.first").show()
      +------+
      |    id|
      +------+
      |second|
      +------+
      

      Same sequence failed in 2.3.1 as well.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Udbhav Agrawal Udbhav Agrawal
            djiangxu Dong Jiang
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment