Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-1716

[Format/JSON] Use string integer value for Decimals in JSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.7.1
    • Fix Version/s: 0.8.0
    • Component/s: C++, Java

      Description

      Suprisingly, Java and C++ integration tests pass after ARROW-1588. This hides a bug, because we're writing decimal values as hex encoded bytes.

      C++ and Java compare that the bytes are the same, but because C++ is interpreting everything as little endian after ARROW-1588 and Java is big endian the numbers these bytes represent will be different in their respective systems.

      I propose that instead of encoding DecimaArray/DecimalVector values as hex encoded bytes, we store the integer as a string when writing Arrow DecimalArray/DecimalVector data to JSON. This will allow us to compare that the bytes have the same meaning in both systems.

      This requires a change to the way Arrow writes JSON.

      Li Jin was extremely helpful in helping me get to the bottom of this.

      cc Li Jin Wes McKinney Jacques Nadeau

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                cpcloud Phillip Cloud
                Reporter:
                cpcloud Phillip Cloud
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: