[SPARK-9315] SparkR DataFrame improvements to be more R-friendly

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Component/s: SparkR
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Shivaram Venkataraman
• Created: 24/Jul/15 18:01
• Updated: 21/May/19 04:33
• Resolved: 21/May/19 04:33

Description

This umbrella will track a bunch of functions that will make SparkR DataFrames more friendly to R users. The goal here is to add functions which make SparkR DataFrames resemble local R data frames better. A similar exercise for dplyr compatibility was done in https://issues.apache.org/jira/browse/SPARK-7231

Sub-Tasks

• Add support for filtering using `[` (synonym for filter / select): Resolved
• Add `merge` as synonym for join: Resolved
• Add support for setting column names, types: Resolved
• Add `summary` as a synonym for `describe`: Resolved
• Add nrow, ncol, dim for SparkR data frames: Resolved
• Add rbind as a synonym for `unionAll`: Resolved
• Add `unique` as a synonym for `distinct`: Resolved
• Support `collect` on DataFrame columns: Resolved
• Add transform and subset to DataFrame: Resolved
• DataFrame show method - show(df) should show first N number of rows, similar to R: Closed
• SparkR: Add sort function to dataframe: Resolved
• SparkR: Add correlation function to dataframe: Resolved
• Method coltypes() to return the R column types of a DataFrame: Resolved
• Add as.DataFrame as a synonym for createDataFrame: Resolved
• Simplify SQLContext method signatures and use a singleton: Resolved
• Add attach() function for DataFrame: Resolved
• SparkR: Add merge to DataFrame: Resolved
• SparkR str() method on DataFrame objects: Resolved
• Improve R context management story and add getOrCreate: Resolved
• SparkR: Documentation change for merge function: Resolved
• Add 'with' API: Resolved
• Add parameter drop to subsetting operator [: Resolved
• Add support for head on DataFrame Column: Resolved