[SPARK-26759] Arrow optimization in SparkR's interoperability

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Done
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: SparkR, SQL
• Labels: release-notes
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 29/Jan/19 05:57
• Updated: 12/Dec/22 18:10
• Resolved: 02/May/19 12:47

Description

Arrow 0.12.0 is release and it contains R API. We could optimize Spark DaraFrame <> R DataFrame interoperability.

For instance see the examples below:

• dapply

    df <- createDataFrame(mtcars)
    collect(dapply(df,
                   function(r.data.frame) {
                     data.frame(r.data.frame$gear)
                   },
                   structType("gear long")))
    

• gapply

    df <- createDataFrame(mtcars)
    collect(gapply(df,
                   "gear",
                   function(key, group) {
                     data.frame(gear = key[[1]], disp = mean(group$disp) > group$disp)
                   },
                   structType("gear double, disp boolean")))
    

• R DataFrame -> Spark DataFrame

    createDataFrame(mtcars)
    

• Spark DataFrame -> R DataFrame

    collect(df)
    head(df)
    

Currently, some of communication path between R side and JVM side has to buffer the data and flush it at once due to ARROW-4512. I don't target to fix it under this umbrella.

Issue Links

• is related to: SPARK-27834 Make separate PySpark/SparkR vectorization configurations
• relates to: SPARK-21187 Complete support for remaining Spark data types in Arrow Converters

Sub-Tasks

• Arrow optimization for conversion from R DataFrame to Spark DataFrame: Resolved
• Arrow optimization for conversion from Spark DataFrame to R DataFrame: Resolved
• Vectorized dapply, Arrow optimization in native R function execution: Resolved
• Vectorized gapply, Arrow optimization in native R function execution: Resolved
• Vectorized gapplyCollect, Arrow optimization in native R function execution: Resolved
• Deduplicate type checking across Arrow optimization and vectorized APIs in SparkR: Resolved
• Document Arrow optimization and vectorized R APIs: Resolved
• Refactor ArrowRRunner and RRunner to share the same base: Resolved
• Set socket timeout consistently in Arrow optimization: Resolved