[ARROW-1422] [Format] Add specification document for the serialization scheme used in Python

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Component/s: Format, Python
• Labels: pyarrow-serialization
• External issue URL: https://github.com/apache/arrow/issues/17447
• Assignee: Unassigned
• Reporter: Wes McKinney
• Created: 28/Aug/17 13:41
• Updated: 11/Jan/23 07:14
• Resolved: 17/Feb/21 16:04

Description

In discussing the new pyarrow.serialize function with attendees at JupyterCon last week, there is some interest in possibly using this serialization scheme (a union with scalar values + object structure, with a sidecar of tensors) beyond Python. For example, it would be interesting to be able to interact with this "data structure" in JavaScript. It may be worth writing down the details of how this works so that other implementations can be created without having to pore through the C++ code for details