[PARQUET-2017] Handle special values of floating point statistics

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: parquet-mr
• Assignee: Gabor Szadovszky
• Reporter: Gabor Szadovszky
• Created: 07/Apr/21 11:40
• Updated: 07/Apr/21 11:40

Description

Based on PARQUET-1251 we have implemented the suggested workaround but it is not complete in all situations.

• We handle the special floating point values at row-group level in parquet-mr but only for the read path. For the write path we still write these values.
• For column indexes we handle these values but only for the write path and not for the read path.

We should implement the workaround for both read and write paths for all cases so we not only handle potentially invalid values but also don't write them to the file.