[SPARK-22228] Add support for Array<primitive_type> so from_json can parse

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.2.0
• Component/s: Java API
• Assignee: Unassigned
• Reporter: kant kodali
• Created: 09/Oct/17 18:46
• Updated: 23/Jul/18 01:18
• Resolved: 23/Jul/18 01:18

Description

val inputDS = Seq("""["foo", "bar"]""").toDF

inputDS.printSchema()
root
 |-- value: string (nullable = true)

Input Dataset inputDS

inputDS.show(false)

value
-----
["foo", "bar"]

Expected output dataset outputDS

value
-------
"foo" |
"bar" |

Tried explode function like below but it doesn't quite work

inputDS.select(explode(from_json(col("value"), ArrayType(StringType))))

and got the following error

org.apache.spark.sql.AnalysisException: cannot resolve 'jsontostructs(`value`)' due to data type mismatch: Input schema string must be a struct or an array of structs

Also tried the following

inputDS.select(explode(col("value")))

And got the following error

org.apache.spark.sql.AnalysisException: cannot resolve 'explode(`value`)' due to data type mismatch: input to function explode should be array or map type, not StringType

Issue Links

• duplicates: SPARK-24391 from_json should support arrays of primitives, and more generally all JSON
• links to: [Github] Pull Request #19492 (mgaido91)