[HIVE-14324] ORC PPD for floats is broken

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.3.0, 2.0.0, 2.1.0, 2.2.0
• Fix Version/s: 2.1.1, 2.2.0
• Component/s: ORC
• Target Version/s: 2.1.0, 2.2.0
• Assignee: Prasanth Jayachandran
• Reporter: Prasanth Jayachandran
• Created: 25/Jul/16 08:23
• Updated: 08/Dec/16 14:15
• Resolved: 26/Jul/16 19:29

Description

ORC stores min/max stats, bloom filters by passing floats as doubles using java's widening conversion. So if we write a float value of 0.22 to ORC file, the min/max stats and bloom filter will use 0.2199999988079071 double value.
But when we do PPD, SARG creates literals by converting float to string and then to double which compares 0.22 to 0.2199999988079071 and fails PPD evaluation.

hive> create table orc_float (f float) stored as orc;
hive> insert into table orc_float values(0.22);
hive> set hive.optimize.index.filter=true;
hive> select * from orc_float where f=0.22;
OK
hive> set hive.optimize.index.filter=false;
hive> select * from orc_float where f=0.22;
OK
0.22

This is not a problem for doubles and decimals.

This issue was introduced in HIVE-8460 but back then there was no strict type check when SARGs are created and also PPD evaluation does not convert to column type. But now predicate leaf creation in SARG enforces strict type check for boxed literals and predicate type and PPD evaluation converts stats and constants to column type (predicate).

Issue Links

• blocks: HIVE-14310 ORC schema evolution should not completely disable PPD