[SPARK-19849] Support ArrayType in to_json function/expression

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.2.0
• Component/s: SQL
• Target Version/s: 2.2.0
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 07/Mar/17 14:42
• Updated: 12/Dec/22 17:51
• Resolved: 20/Mar/17 05:39

Description

After SPARK-19595, we could

import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
val schema = ArrayType(StructType(StructField("a", IntegerType) :: Nil))
Seq(("""[{"a": 1}, {"a": 2}]""")).toDF("array").select(from_json(col("array"), schema)).show()

Maybe, it'd be better if we provide a way to read it back via to_json as below:

import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._

val schema = ArrayType(StructType(StructField("a", IntegerType) :: Nil))
val df = Seq("""[{"a":1}, {"a":2}]""").toDF("json").select(from_json($"json", schema).as("array"))
df.show()

// Read back.
df.select(to_json($"array").as("json")).show()

+----------+
|     array|
+----------+
|[[1], [2]]|
+----------+

+-----------------+
|             json|
+-----------------+
|[{"a":1},{"a":2}]|
+-----------------+

Currently, it throws an exception as below:

org.apache.spark.sql.AnalysisException: cannot resolve 'structtojson(`array`)' due to data type mismatch: structtojson requires that the expression is a struct expression.;;
'Project [structtojson(array#30, Some(Asia/Seoul)) AS structtojson(array)#45]
+- Project [jsontostruct(ArrayType(StructType(StructField(a,IntegerType,true)),true), array#27, Some(Asia/Seoul)) AS array#30]
   +- Project [value#25 AS array#27]
      +- LocalRelation [value#25]

  at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:80)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:72)

Issue Links

• relates to: SPARK-19595 from_json produces only a single row when input is a json array
• links to: [Github] Pull Request #17192 (HyukjinKwon)