[ARROW-14379] [R] Create a custom extension of list that stores row-level metadata

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Component/s: R
• External issue URL: https://github.com/apache/arrow/issues/29947
• Assignee: Unassigned
• Reporter: Jonathan Keane
• Created: 19/Oct/21 13:29
• Updated: 11/Jan/23 08:39
• Resolved: 26/Oct/22 20:29

Description

Since lists can be nested, we should be able store each element as something like list(value = "foo", attributes = list(attr1 = TRUE, attr2 = "baz")) and then we can reconstitute that in the R conversion to transfer the attributes element to attributes.

This will be more efficient (since we get compression on the column + metadata/attributes) and we also will be able to filter these + use them in datasets since each row has all of the information about itself that it needs to roundtrip.

This would get us SF columns for free