[ORC-629] PPD: Floating point NaN is not transitive across comparisons

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.4, 1.4.5, 1.6.3, 1.7.0
• Fix Version/s: 1.6.4, 1.7.0
• Component/s: Java, Reader
• Assignee: Panagiotis Garefalakis
• Reporter: Gopal Vijayaraghavan
• Created: 07/May/20 00:12
• Updated: 05/Nov/20 14:40
• Resolved: 14/May/20 18:58

Description

Range comparisons don't work right for columns which start with Double.NaN as the first row (min == max == NaN).

1 < NaN is false.
1 > NaN is false.

File Version: 0.12 with ORC_135
Rows: 3
Compression: ZLIB
Compression size: 32768
Type: struct<operation:int,originalTransaction:bigint,bucket:int,rowId:bigint,currentTransaction:bigint,row:struct<c:double>>

Stripe Statistics:
  Stripe 1:
    Column 0: count: 3 hasNull: false
    Column 1: count: 3 hasNull: false bytesOnDisk: 5 min: 0 max: 0 sum: 0
    Column 2: count: 3 hasNull: false bytesOnDisk: 5 min: 1 max: 1 sum: 3
    Column 3: count: 3 hasNull: false bytesOnDisk: 8 min: 536870912 max: 536870912 sum: 1610612736
    Column 4: count: 3 hasNull: false bytesOnDisk: 7 min: 0 max: 2 sum: 3
    Column 5: count: 3 hasNull: false bytesOnDisk: 5 min: 1 max: 1 sum: 3
    Column 6: count: 3 hasNull: false
    Column 7: count: 3 hasNull: false bytesOnDisk: 19 min: NaN max: NaN sum: NaN

Issue Links

• is a parent of: ORC-683 PPD: Make Floating point NaN check more strict
• is related to: PARQUET-1246 Ignore float/double statistics in case of NaN
• is related to: ORC-636 [C++] PPD Floating point stats with NaN should be ignored
• links to: Github PR
• links to: GitHub Pull Request #508