[ARROW-3263] [R] Use R sentinel values for missingness in addition to bitmask

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Format, R
• External issue URL: https://github.com/apache/arrow/issues/19603
• Assignee: Unassigned
• Reporter: Gabriel Becker
• Created: 18/Sep/18 17:29
• Updated: 11/Jan/23 07:26

Description

R uses sentinal values to indicate missingness within Atomic vectors (read arrays in Arrow parlance, AFAIK). 

Currently according to wesmckinn, the current value in the array in memory is undefined if the bitmap indicating missingness is set to 1. 

This will force R to copy and modify data whenever adopting Arrow data which has missingness present as a native vector.

If the value were written to the relevant sentinal values (INT_MIN for 32 bit integers, and NaN with payload 1954 for double precision floats) in addition to the bit mask, then R would be able to use Arrow as intended while not breaking any other systems.

Issue Links

• is related to: ARROW-8348 [C++] Support optional sentinel values in primitive Array for nulls