Uploaded image for project: 'ORC'
  1. ORC
  2. ORC-452

Support converting MAP column from JSON to ORC

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.5.5, 1.6.0
    • Component/s: Java, tools
    • Labels:
      None

      Description

      The JSON convert tool has not supported MAP column yet.

      java -jar orc-tools-1.5.4-uber.jar convert -s "struct<id:int,map_col:map<string,string>>" map.json 
      log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
      log4j:WARN Please initialize the log4j system properly.
      log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
      Processing map.json
      Exception in thread "main" java.lang.IllegalArgumentException: Unhandled type map<string,string>
              at org.apache.orc.tools.convert.JsonReader.createConverter(JsonReader.java:252)
              at org.apache.orc.tools.convert.JsonReader.<init>(JsonReader.java:277)
              at org.apache.orc.tools.convert.JsonReader.<init>(JsonReader.java:260)
              at org.apache.orc.tools.convert.ConvertTool$FileInformation.getRecordReader(ConvertTool.java:149)
              at org.apache.orc.tools.convert.ConvertTool.run(ConvertTool.java:203)
              at org.apache.orc.tools.convert.ConvertTool.main(ConvertTool.java:165)
              at org.apache.orc.tools.Driver.main(Driver.java:113)
      

      In JsonReader.java, there's no converter for MAP type.

      We'd like to convert following JSON rows into an ORC file

      {"id": 0, "map_col": {"k1": "v1", "k2": "v2"}}
      {"id": 1, "map_col": {"k3": "v3", "k4": "v4", "k5": "v5"}}

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                stigahuang Quanlong Huang
                Reporter:
                stigahuang Quanlong Huang
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m