[HIVE-23036] ORC PPD eval with sub-millisecond timestamps

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Labels: pull-request-available
• Assignee: Panagiotis Garefalakis
• Reporter: Panagiotis Garefalakis
• Created: 17/Mar/20 14:16
• Updated: 17/Nov/22 08:50
• Resolved: 08/Feb/21 18:30

Description

See ORC-611 for more details

ORC stores timestamps with:

• nanosecond precision for the data itself
• milliseconds precision for min-max statistics

As both min and max are rounded to the same value,  timestamps with ns precision will not pass the PPD evaluator.

create table tsstat (ts timestamp) stored as orc;
insert into tsstat values ("1970-01-01 00:00:00.0005");
select * from tsstat where ts = "1970-01-01 00:00:00.0005";
-- returned 0 rows

ORC PPD evaluation currently happens as part of OrcInputFormat https://github.com/apache/hive/blob/7e39a2c13711f9377c9ce1edb4224880421b1ea5/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java#L2314

Issue Links

• is related to: ORC-611 Incorrect min-max stats for sub-millisecond timestamps
• links to: GitHub Pull Request #1943