[SPARK-32694] Pushdown cast to data sources

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.0.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Chao Sun
• Created: 24/Aug/20 20:48
• Updated: 04/Dec/20 20:49
• Resolved: 26/Aug/20 04:30

Description

Currently we don't support pushing down cast to data source (see link for a discussion). For instance, in the following code snippet:

scala> case class Person(name: String, age: Short)
scala> Seq(Person("John", 32), Person("David", 25), Person("Mike", 18)).toDS().write.parquet("/tmp/person.parquet")
scala> val personDS = spark.read.parquet("/tmp/person.parquet")
scala> personDS.createOrReplaceTempView("person")
scala> spark.sql("SELECT * FROM person where age < 30")

The predicate won't be pushed down to Parquet data source because in DataSourceStrategy, PushableColumnBase only handles a few limited cases such as Attribute and GetStructField. Potentially we can handle Cast here as well.

Issue Links

• is duplicated by: SPARK-24994 Add UnwrapCastInBinaryComparison optimizer to simplify literal types