[SPARK-33795] gapply fails execution with rbind error

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 3.0.0
• Component/s: SparkR

Environment

Databricks runtime 7.3 LTS ML

• Assignee: Unassigned
• Reporter: MvR
• Created: 15/Dec/20 11:50
• Updated: 12/Dec/22 18:11
• Resolved: 21/Dec/20 10:22

Description

Executing following code on databricks runtime 7.3 LTS ML errors out showing some rbind error whereas it is successfully executed without enabling Arrow in Spark session. Full error message attached.

 

```

library(dplyr)
library(SparkR)

SparkR::sparkR.session(sparkConfig = list(spark.sql.execution.arrow.sparkr.enabled = "true"))

mtcars %>%
SparkR::as.DataFrame() %>%

SparkR::gapply(x = .,
cols = c("cyl", "vs"),

func = function(key,
data)

{ dt <- data[,c("mpg", "qsec")] res <- apply(dt, 2, mean) df <- data.frame(firstGroupKey = key[1], secondGroupKey = key[2], mean_mpg = res[1], mean_cyl = res[2]) return(df) }

,
schema = structType(structField("cyl", "double"),
structField("vs", "double"),
structField("mpg_mean", "double"),
structField("qsec_mean", "double"))
) %>%
display()

```