Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21513

SQL to_json should support all column types

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.2.0
    • 2.3.0
    • SQL

    Description

      The built-in SQL UDF "to_json" currently supports serializing StructType columns, as well as Arrays of StructType columns. If you attempt to use it on a different type, for example a map, you get an error like this:

      AnalysisException: cannot resolve 'structstojson(`tags`)' due to data type mismatch: Input type map<string,string> must be a struct or array of structs.;;
      

      This limitation seems arbitrary; if I were to go through the effort of enclosing my map in a struct, it would be serializable. Same thing with any other non-struct type.

      Therefore the desired improvement is to allow to_json to operate directly on any column type. The associated code is here.

      Attachments

        Activity

          People

            goldmedal Jia-Xuan Liu
            ilikerps Aaron Davidson
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: