[CALCITE-6395] Significant precision loss when representing REAL literals

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.36.0
• Fix Version/s: 1.38.0
• Component/s: core
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Mihai Budiu
• Created: 01/May/24 23:00
• Updated: 17/Oct/24 22:41
• Resolved: 17/Oct/24 22:41

Description

Consider this test that could be a SqlOperatorTest:

    f.checkScalar("CAST(CAST('36854775807.0' AS REAL) AS BIGINT)",
        "36854775808", "BIGINT NOT NULL");

The produced result is actually very far:

Expected: is "36854775808"
but: was "36854779904"

This big error comes from two reasons:

• Calcite uses BigDecimal values to represent floating point values, see CALCITE-2067
• When converting a Float value to a BigDecimal in RexBuilder.clean(), the following sequence is used:

new BigDecimal(((Number) o).doubleValue(), MathContext.DECIMAL32)

Using a DECIMAL32 math context leads to the precision loss. Just because a Float uses 32 bits does not mean that the decimal should also use 32 bits.

The real fix is in the PR for CALCITE-2067, but that hasn't been reviewed for a long time, so I will submit a fix for the clean() method..

Issue Links

• links to: GitHub Pull Request #3782