[ORC-452] Support converting MAP column from JSON to ORC

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.5, 1.6.0
• Component/s: Java, tools
• Assignee: Quanlong Huang
• Reporter: Quanlong Huang
• Created: 25/Dec/18 13:06
• Updated: 23/Mar/21 16:35
• Resolved: 09/Mar/19 00:32

Description

The JSON convert tool has not supported MAP column yet.

java -jar orc-tools-1.5.4-uber.jar convert -s "struct<id:int,map_col:map<string,string>>" map.json 
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Processing map.json
Exception in thread "main" java.lang.IllegalArgumentException: Unhandled type map<string,string>
        at org.apache.orc.tools.convert.JsonReader.createConverter(JsonReader.java:252)
        at org.apache.orc.tools.convert.JsonReader.<init>(JsonReader.java:277)
        at org.apache.orc.tools.convert.JsonReader.<init>(JsonReader.java:260)
        at org.apache.orc.tools.convert.ConvertTool$FileInformation.getRecordReader(ConvertTool.java:149)
        at org.apache.orc.tools.convert.ConvertTool.run(ConvertTool.java:203)
        at org.apache.orc.tools.convert.ConvertTool.main(ConvertTool.java:165)
        at org.apache.orc.tools.Driver.main(Driver.java:113)

In JsonReader.java, there's no converter for MAP type.

We'd like to convert following JSON rows into an ORC file

{"id": 0, "map_col": {"k1": "v1", "k2": "v2"}}
{"id": 1, "map_col": {"k3": "v3", "k4": "v4", "k5": "v5"}}

 

Issue Links

• is related to: ORC-726 Support Map type in `orc-tools convert`
• links to: GitHub Pull Request #354