Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Implemented
-
None
-
None
-
None
Description
currently in SparkR, collect() on a DataFrame collects the data within the DataFrame into a local data.frame. R users are used to using data.frame.
However, collect() currently can't collect data of nested types from a DataFrame because:
1. The serializer in JVM backend does not support nested types;
2. collect() in R side assumes each column is of simple atomic type that can be combinded into a atomic vector
Attachments
Attachments
Issue Links
- is related to
-
SPARK-10048 Support arbitrary nested Java array in serde
- Resolved
-
SPARK-6819 Support nested types in SparkR DataFrame
- Resolved