[ARROW-56] Format: Specify LSB bit ordering in bit arrays

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.1.0
• Component/s: Format
• External issue URL: https://github.com/apache/arrow/issues/15467
• Assignee: Wes McKinney
• Reporter: Wes McKinney
• Created: 10/Mar/16 00:36
• Updated: 11/Jan/23 07:06
• Resolved: 25/Mar/16 02:21

Description

In https://github.com/apache/arrow/blob/master/format/Layout.md, we describe bitmaps for nulls (presumably same encoding for boolean data) and provide the C code for accessing a bit:

bitmask[j / 8] & (1 << (j % 8))

Implicitly this is LSB bit-endianness (https://en.wikipedia.org/wiki/Bit_numbering) – see the discussion of bitpacking in the Parquet format: https://github.com/apache/parquet-format/blob/master/Encodings.md#run-length-encoding--bit-packing-hybrid-rle--3

If this is what we want, we should make this aspect of the format explicit.