Details
-
New Feature
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
Description
After many mailing list discussions and a first draft of metadata serialization (https://github.com/apache/arrow/blob/master/format/Message.fbs), I propose we strike the notion of required fields from the physical memory layout and instead make this a property of the metadata (see the nullable member of Field in the serialized schema metadata).
Note that Arrow implementations will be free to create specialized containers for data having nullable = false in their metadata. It will be the implementations' responsibility to validate that null_count == 0 at IPC boundaries when interacting with "untrusted" data.