Uploaded image for project: 'Calcite'
  1. Calcite
  2. CALCITE-972

Decimal data sent by Avatica cannot be interpreted by some clients

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.5.0
    • None
    • avatica

    Description

      JSON specifies a 'number' data type which does not differentiate between integer and floating-point values. The encoding for JSON numbers is implementation-specific. Libraries such as RapidJSON can treat all non-integer values as 'double' while still being compliant with JSON specifications. As a result, decimal values sent by Phoenix will lose precision on the client side.

      Decimal data should be sent as a string. If it is sent as a number, we cannot be expect that all JSON-spec compliant library will handle it properly.

      ie. Phoenix sends a decimal as a JSON number, but Rapidjson can only extract JSON numbers as integer or double because the C++ language does not have native decimal support.

      Attachments

        Issue Links

          Activity

            elserj Josh Elser added a comment - - edited

            FWIW, I ended up serializing decimals as strings w/ protobuf serialization in CALCITE-1103. Yay for consistent ideas.

            I'm leaving this open, though, for the Rep.STRING and DecimalFromStringAccessor changes that Julian mentioned.

            elserj Josh Elser added a comment - - edited FWIW, I ended up serializing decimals as strings w/ protobuf serialization in CALCITE-1103 . Yay for consistent ideas. I'm leaving this open, though, for the Rep.STRING and DecimalFromStringAccessor changes that Julian mentioned.
            kliew Kevin Liew added a comment -

            That sounds good. We don't require scientific notation, and any library that can interpret scientific notation should also be able to read a simple decimal string.
            Your proposed format should be compatible with all clients.

            kliew Kevin Liew added a comment - That sounds good. We don't require scientific notation, and any library that can interpret scientific notation should also be able to read a simple decimal string. Your proposed format should be compatible with all clients.
            julianhyde Julian Hyde added a comment - - edited

            Makes sense. Let's agree on the format of the string. Hopefully it can be something simple, e.g.

            /-?[0-9]+(\.[0-9]+)?/

            We don't need scientific notation, right?

            The client should be expecting strings - the column should be declared as using Rep.STRING - and can therefore create a DecimalFromStringAccessor.

            julianhyde Julian Hyde added a comment - - edited Makes sense. Let's agree on the format of the string. Hopefully it can be something simple, e.g. /-?[0-9]+(\.[0-9]+)?/ We don't need scientific notation, right? The client should be expecting strings - the column should be declared as using Rep.STRING - and can therefore create a DecimalFromStringAccessor.

            People

              Unassigned Unassigned
              kliew Kevin Liew
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated: