[SPARK-10047] Improve the implementation of collect() on DataFrame in SparkR

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Implemented
• Component/s: SparkR
• Assignee: Unassigned
• Reporter: Sun Rui
• Created: 17/Aug/15 07:11
• Updated: 22/Sep/15 08:21
• Resolved: 22/Sep/15 08:21

Description

currently in SparkR, collect() on a DataFrame collects the data within the DataFrame into a local data.frame. R users are used to using data.frame.

However, collect() currently can't collect data of nested types from a DataFrame because:
1. The serializer in JVM backend does not support nested types;
2. collect() in R side assumes each column is of simple atomic type that can be combinded into a atomic vector

Issue Links

• is related to: SPARK-10048 Support arbitrary nested Java array in serde
• is related to: SPARK-6819 Support nested types in SparkR DataFrame