[KUDU-2821] Push down string range filter from Spark on timestamp column

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.9.0
• Component/s: spark
• Assignee: Unassigned
• Reporter: Jacob Evan Beard
• Created: 09/May/19 21:57
• Updated: 09/May/19 22:53

Description

In the Spark-Kudu integration, a range filter on a Kudu timestamp column using a string representation will not be pushed down to Kudu. It would be good if it did!

For example:

df.where("time > '2019-05-09 19:04:17' and time < '2019-05-09 19:04:26'").count

will not push down to Kudu, but

df.where("time > cast('2019-05-09 19:04:17' as timestamp) and time < cast('2019-05-09 19:04:26' as timestamp)").count

will push down to Kudu.