[SPARK-27160] Incorrect Literal Casting of DecimalType in OrcFilters

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0, 2.4.0
• Fix Version/s: 2.3.4, 2.4.1, 3.0.0
• Component/s: SQL
• Labels: correctness
• Assignee: Darcy Shen
• Reporter: Darcy Shen
• Created: 14/Mar/19 13:27
• Updated: 02/Mar/20 20:41
• Resolved: 20/Mar/19 19:32

Description

DecimalType Literal should not be casted to Long.

eg. For `df.filter("x < 3.14")`, assuming df (x in DecimalType) reads from a ORC table and uses the native ORC reader with predicate push down enabled, we will push down the `x < 3.14` predicate to the ORC reader via a SearchArgument.

OrcFilters will construct the SearchArgument, but not handle the DecimalType correctly.

The previous impl will construct `x < 3` from `x < 3.14`.

Issue Links

• links to: GitHub Pull Request #24092