[SPARK-6819] Support nested types in SparkR DataFrame

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.6.0
• Component/s: SparkR, SQL
• Target Version/s: 1.6.0
• External issue URL: https://sparkr.atlassian.net/browse/SPARKR-230
• Assignee: Sun Rui
• Reporter: Shivaram Venkataraman
• Created: 09/Apr/15 21:03
• Updated: 03/Nov/15 16:27
• Resolved: 03/Nov/15 16:27

Description

ArrayType, MapType and StructureType
We can infer the correct schema for them, but the serialization can not handle them well.

From Davies in https://sparkr.atlassian.net/browse/SPARKR-230

ArrayType could be c("ab", "ba"), we will serialize it as character. We does not support deserialize env in R (serialize Map[] in Scala). So all the three do not work.

Issue Links

• blocks: SPARK-9972 Add `struct`, `encode` and `decode` function in SparkR
• relates to: SPARK-10047 Improve the implementation of collect() on DataFrame in SparkR

Sub-Tasks

• Support arbitrary nested Java array in serde: Resolved
• Support collecting data of ArraryType in DataFrame: Resolved
• Support collecting data of MapType in DataFrame: Resolved
• Support collecting data of StructType in DataFrame: Resolved