[ARROW-786] [Format] In-memory format for 128-bit Decimals, handling of sign bit

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.7.0
• Component/s: Format
• External issue URL: https://github.com/apache/arrow/issues/16398
• Assignee: Phillip Cloud
• Reporter: Wes McKinney
• Created: 07/Apr/17 15:51
• Updated: 11/Jan/23 07:10
• Resolved: 24/Aug/17 17:57

Description

cc cpcloud

We found in ARROW-655 that we needed to add an extra bit for signedness for decimals stored as 128-bit values to be able to use the Boost multiprecision libraries. This makes Decimal128 not fit completely neatly as a 16-byte fixed size binary value, and more of a struct<sign_bitmap: boolean, data: fixed_size_binary(16)>. What is the current formata in the Java implementation? We will need to document the memory layout for decimals that maximizes compatibility across languages and eventually implement integration tests for IPC.

Issue Links

• blocks: ARROW-695 Integration tests for Decimal types
• is related to: SPARK-21552 Add decimal type support to ArrowWriter.