[ARROW-8952] [C++] Support for textual, JSON schema representation

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: C++
• Labels: Schema pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/25078
• Assignee: Unassigned
• Reporter: Christian Hudon
• Created: 26/May/20 18:04
• Updated: 11/Jan/23 08:03

Description

Currently, Arrow has no textual representation for its schema that could serve the same purposes as JSON-Schema for JSON, the .proto files for Protobuf, etc. This issue is about adding such a text representation for an Arrow schema, to fill the same use cases that these textual representations fill for other data serialization formats.

The requirements for a text schema representation:

• Data, not code (can be used without being run directly, unlike e.g. calls to the Python API to create a Schema object)
• Readable by people who are experts in their field (e.g. data scientists, etc.) and are however not Arrow experts, without needing the doc side by side
• Small modifications possible with no or light usage of the doc (e.g. changing a field from int32 to int64)
• Writing new schemas from scratch possible with the doc for non-Arrow experts
• Not tied to a particular version of Arrow & compatible between Arrow versions

And from a software engineering point of view, it would be very desirable for the implementation to not add another library dependency for Arrow (which already has many).

After discussion on the mailing list, the JSON representation for Flatbuffers data seemed the best candidate. It is a format supported by the Flatbuffers projects for serializing Flatbuffers assets in a human-readable format, for inclusion under source-control. And there is already functionality in Arrow to convert Schema objects to a Flatbuffers representation. This would meet all the requirements above, while requiring only a small amount of new Arrow code to implement.

This issue will add functions Arrow to load and save a textual, JSON representation of an Arrow schema, by first converting it to a FlatBuffers object, and then using the Flatbuffers functionality to save/load such objects as JSON.

Issue Links

• supercedes: ARROW-17998 [Java] Support for textual JSON schema representation (was: JSON representation of pojo.Schema is incompatible with flatbuffers JSON generated via C++ API)
• links to: GitHub Pull Request #7110
• links to: WIP Pull Request