[PARQUET-1880] Convert JSON to Parquet directly

• Type: Wish
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: parquet-mr
• Assignee: Unassigned
• Reporter: Emmanuel Brard
• Created: 23/Jun/20 12:36
• Updated: 26/Jun/20 14:42

Description

In order to convert JSON payload to Parquet, the "established" way seems to be to go first through a JSON to Avro conversion, provided the JSON payload is covered by a JSON-Schema or part of a OpenAPI definition. The rest of the process relies on the parquet-avro implementation.

Since JSON payload can be loaded to a generic JsonNode (jackson library) and since a JSON payload can be described and checked against a JSON-Schema or an OpenAPI definition, wouldn't it be theoretically possible to convert a JSON payload to a Parquet "record" directly (somehow implementing a similar approach than parquet-avro but this a JsonNode as a data holder)?