[SPARK-13207] _SUCCESS should not break partition discovery

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.6.2, 2.0.0
• Component/s: SQL
• Labels: backport-needed
• Target Version/s: 1.6.2, 2.0.0
• Assignee: Yin Huai
• Reporter: Yin Huai
• Created: 05/Feb/16 02:21
• Updated: 11/Jun/16 06:21
• Resolved: 14/Mar/16 16:03

Description

Partitioning discovery will fail with the following case

test("_SUCCESS should not break partitioning discovery") {
    withTempPath { dir =>
      val tablePath = new File(dir, "table")
      val df = (1 to 3).map(i => (i, i, i, i)).toDF("a", "b", "c", "d")

      df.write
        .format("parquet")
        .partitionBy("b", "c", "d")
        .save(tablePath.getCanonicalPath)

      Files.touch(new File(s"${tablePath.getCanonicalPath}/b=1", "_SUCCESS"))
      Files.touch(new File(s"${tablePath.getCanonicalPath}/b=1/c=1", "_SUCCESS"))
      Files.touch(new File(s"${tablePath.getCanonicalPath}/b=1/c=1/d=1", "_SUCCESS"))
      checkAnswer(sqlContext.read.format("parquet").load(tablePath.getCanonicalPath), df)
    }
  }

Because _SUCCESS is the in the inner partitioning dirs, partitioning discovery will fail.

Issue Links

• relates to: SPARK-15895 _common_metadata and _metadata appearing in the inner partitioning dirs of a partitioned parquet datasets break partitioning discovery
• links to: [Github] Pull Request #11088 (yhuai)
• links to: [Github] Pull Request #11697 (yhuai)