Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24027

Support MapType(StringType, DataType) as root type by from_json

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.3.0
    • 2.4.0
    • SQL
    • None

    Description

      Currently, MapType is not supported by the from_json function as the root type. For example, the following code doesn't work on Spark 2.3:

      scala> import org.apache.spark.sql.types._
      import org.apache.spark.sql.types._
      
      scala> val schema = MapType(StringType, IntegerType)
      schema: org.apache.spark.sql.types.MapType = MapType(StringType,IntegerType,true)
      
      scala> val in = Seq("""{"a": 1, "b": 2, "c": 3}""").toDS()
      in: org.apache.spark.sql.Dataset[String] = [value: string]
      
      scala> in.select(from_json($"value", schema, Map[String, String]())).collect()
      org.apache.spark.sql.AnalysisException: cannot resolve 'jsontostructs(`value`)' due to data type mismatch: Input schema map<string,int> must be a struct or an array of structs.
      

      Purpose of the ticket is to support MapType with StringType as keys type.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            maxgekk Max Gekk
            maxgekk Max Gekk
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment