[SPARK-10829] Scan DataSource with predicate expression combine partition key and attributes doesn't work

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.5.2, 1.6.0
• Component/s: SQL
• Target Version/s: 1.5.2, 1.6.0
• Assignee: Cheng Hao
• Reporter: Cheng Hao
• Created: 25/Sep/15 05:58
• Updated: 30/Oct/15 19:16
• Resolved: 14/Oct/15 23:29

Description

To reproduce that with the code:

withSQLConf(SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED.key -> "true") {
      withTempPath { dir =>
        val path = s"${dir.getCanonicalPath}/part=1"
        (1 to 3).map(i => (i, i.toString)).toDF("a", "b").write.parquet(path)

        // If the "part = 1" filter gets pushed down, this query will throw an exception since
        // "part" is not a valid column in the actual Parquet file
        checkAnswer(
          sqlContext.read.parquet(path).filter("a > 0 and (part = 0 or a > 1)"),
          (2 to 3).map(i => Row(i, i.toString, 1)))
      }
    }

We expect the result as:

2, 1
3, 1

But we got:

1, 1
2, 1
3, 1

Issue Links

• links to: [Github] Pull Request #8916 (chenghao-intel)
• links to: [Github] Pull Request #9370 (cloud-fan)
• links to: [Github] Pull Request #9371 (cloud-fan)