Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-29777

SparkR::cleanClosure aggressively removes a function required by user function

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.2, 2.1.3, 2.2.3, 2.3.4, 2.4.4
    • Fix Version/s: 3.0.0
    • Component/s: SparkR
    • Labels:
      None

      Description

      Following code block reproduces the issue:

      df <- createDataFrame(data.frame(x=1))
      f1 <- function(x) x + 1
      f2 <- function(x) f1(x) + 2
      
      dapplyCollect(df, function(x) { f1(x); f2(x) })
      

      We get following error message:

      org.apache.spark.SparkException: R computation failed with
       Error in f1(x) : could not find function "f1"
      Calls: compute -> computeFunc -> f2
      

      Compare that to this code block with succeeds:

      dapplyCollect(df, function(x) { f2(x) })
      

        Attachments

          Activity

            People

            • Assignee:
              falaki Hossein Falaki
              Reporter:
              falaki Hossein Falaki
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: