[SPARK-26199] Long expressions cause mutate to fail

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 3.1.0, 3.2.0
• Component/s: SparkR
• Assignee: Michael Chirico
• Reporter: João Rafael
• Created: 28/Nov/18 12:37
• Updated: 12/Dec/22 18:11
• Resolved: 17/Dec/20 08:22

Description

Calling mutate(df, field = expr) fails when expr is very long.

Example:

df <- mutate(df, field = ifelse(
    lit(TRUE),
    lit("A"),
    ifelse(
        lit(T),
        lit("BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB"),
        lit("C")
    )
))

Stack trace:

FATAL subscript out of bounds
  at .handleSimpleError(function (obj) 
{
    level = sapply(class(obj), sw
  at FUN(X[[i]], ...)
  at lapply(seq_along(args), function(i) {
    if (ns[[i]] != "") {

at lapply(seq_along(args), function(i) {
    if (ns[[i]] != "") {

at mutate(df, field = ifelse(lit(TRUE), lit("A"), ifelse(lit(T), lit("BBB
  at #78: mutate(df, field = ifelse(lit(TRUE), lit("A"), ifelse(lit(T

The root cause is in: DataFrame.R#LL2182

When the expression is long deparse returns multiple lines, causing args to have more elements than ns. The solution could be to set nlines = 1 or to collapse the lines together.

A simple work around exists, by first placing the expression in a variable and using it instead:

tmp <- ifelse(
    lit(TRUE),
    lit("A"),
    ifelse(
        lit(T),
        lit("BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB"),
        lit("C")
    )
)
df <- mutate(df, field = tmp)

Issue Links

• duplicates: SPARK-31517 SparkR::orderBy with multiple columns descending produces error
• is duplicated by: SPARK-31517 SparkR::orderBy with multiple columns descending produces error
• links to: [Github] Pull Request #28386 (MichaelChirico)