Uploaded image for project: 'Calcite'
  1. Calcite
  2. CALCITE-6395

Significant precision loss when representing REAL literals

    XMLWordPrintableJSON

Details

    Description

      Consider this test that could be a SqlOperatorTest:

          f.checkScalar("CAST(CAST('36854775807.0' AS REAL) AS BIGINT)",
              "36854775808", "BIGINT NOT NULL");
      

      The produced result is actually very far:

      Expected: is "36854775808"
      but: was "36854779904"

      This big error comes from two reasons:

      • Calcite uses BigDecimal values to represent floating point values, see CALCITE-2067
      • When converting a Float value to a BigDecimal in RexBuilder.clean(), the following sequence is used:
      new BigDecimal(((Number) o).doubleValue(), MathContext.DECIMAL32)
      

      Using a DECIMAL32 math context leads to the precision loss. Just because a Float uses 32 bits does not mean that the decimal should also use 32 bits.

      The real fix is in the PR for CALCITE-2067, but that hasn't been reviewed for a long time, so I will submit a fix for the clean() method..

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              mbudiu Mihai Budiu
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: