[SPARK-24718] Timestamp support pushdown to parquet data source

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.0
• Component/s: SQL
• Assignee: Yuming Wang
• Reporter: Yuming Wang
• Created: 02/Jul/18 15:45
• Updated: 12/Dec/22 18:11
• Resolved: 15/Jul/18 03:14

Description

Some thing like this:

case ParquetSchemaType(TIMESTAMP_MICROS, INT64, null)
  if pushDownDecimal =>
  (n: String, v: Any) => FilterApi.eq(
    longColumn(n),
    Option(v).map(t => (t.asInstanceOf[java.sql.Timestamp].getTime * 1000)
      .asInstanceOf[java.lang.Long]).orNull)
case ParquetSchemaType(TIMESTAMP_MILLIS, INT64, null)
  if pushDownDecimal =>
  (n: String, v: Any) => FilterApi.eq(
    longColumn(n),
    Option(v).map(_.asInstanceOf[java.sql.Timestamp].getTime
      .asInstanceOf[java.lang.Long]).orNull)

Issue Links

• is duplicated by: SPARK-11784 Support Timestamp filter pushdown in Parquet datasource
• is related to: SPARK-24716 Refactor ParquetFilters
• links to: [Github] Pull Request #21741 (wangyum)