Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-16785

dapply doesn't return array or raw columns

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.0.0
    • 2.0.1, 2.1.0
    • SparkR
    • None
    • Mac OS X

    Description

      Calling SparkR::dapplyCollect with R functions that return dataframes produces an error. This comes up when returning columns of binary data- ie. serialized fitted models. Also happens when functions return columns containing vectors.

      The error message:

      R computation failed with
      Error in (function (..., deparse.level = 1, make.row.names = TRUE, stringsAsFactors = default.stringsAsFactors()) :
      invalid list argument: all variables should have the same length

      Reproducible example: https://github.com/clarkfitzg/phd_research/blob/master/ddR/spark/sparkR_dapplyCollect7.R

      Relates to SPARK-16611

      Attachments

        Activity

          People

            clarkfitzg Clark Fitzgerald
            clarkfitzg Clark Fitzgerald
            Shivaram Venkataraman Shivaram Venkataraman
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: