[SPARK-9302] Handle complex JSON types in collect()/head()

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.0, 1.4.1
• Component/s: SparkR
• Assignee: Unassigned
• Reporter: Sun Rui
• Created: 24/Jul/15 05:28
• Updated: 14/Oct/15 04:32
• Resolved: 14/Oct/15 04:32

Description

Reported in the mailing list by Exie <[e-mail]>:

A sample record in raw JSON looks like this:
{"version": 1,"event": "view","timestamp": 1427846422377,"system":
"DCDS","asset": "6404476","assetType": "myType","assetCategory":
"myCategory","extras": [{"name": "videoSource","value": "mySource"},{"name":
"playerType","value": "Article"},{"name": "duration","value":
"202088"}],"trackingId": "155629a0-d802-11e4-13ee-6884e43d6000","ipAddress":
"165.69.2.4","title": "myTitle"}

> head(mydf)
Error in as.data.frame.default(x[[i]], optional = TRUE) : 
  cannot coerce class ""jobj"" to a data.frame
>
> show(mydf)
DataFrame[localEventDtTm:timestamp, asset:string, assetCategory:string, assetType:string, event:string, extras:array<struct&lt;name:string,value:string>>, ipAddress:string, memberId:string, system:string, timestamp:bigint, title:string, trackingId:string, version:bigint]
>

Issue Links

• is related to: SPARK-10049 Support collecting data of ArraryType in DataFrame
• is related to: SPARK-10051 Support collecting data of StructType in DataFrame