Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-17577

SparkR support add files to Spark job and get by executors

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.1, 2.1.0
    • Component/s: SparkR
    • Labels:
      None

      Description

      Scala/Python users can add files to Spark job by submit options --files or SparkContext.addFile(). Meanwhile, users can get the added file by SparkFiles.get(filename).
      We should also support this function for SparkR users, since they also have the requirements for some shared dependency files. For example, SparkR users can download third party R packages to driver firstly, add these files to the Spark job as dependency by this API and then each executor can install these packages by install.packages.

        Issue Links

          Activity

          Hide
          apachespark Apache Spark added a comment -

          User 'yanboliang' has created a pull request for this issue:
          https://github.com/apache/spark/pull/15131

          Show
          apachespark Apache Spark added a comment - User 'yanboliang' has created a pull request for this issue: https://github.com/apache/spark/pull/15131
          Hide
          apachespark Apache Spark added a comment -

          User 'yanboliang' has created a pull request for this issue:
          https://github.com/apache/spark/pull/15216

          Show
          apachespark Apache Spark added a comment - User 'yanboliang' has created a pull request for this issue: https://github.com/apache/spark/pull/15216
          Hide
          apachespark Apache Spark added a comment -

          User 'yanboliang' has created a pull request for this issue:
          https://github.com/apache/spark/pull/15217

          Show
          apachespark Apache Spark added a comment - User 'yanboliang' has created a pull request for this issue: https://github.com/apache/spark/pull/15217
          Hide
          sarutak Kousuke Saruta added a comment -

          I add 2.0.1 as a fix version but we are in voting for 2.0.1 (RC2) now. If RC2 passes, I'll modified the version to 2.0.2.

          Show
          sarutak Kousuke Saruta added a comment - I add 2.0.1 as a fix version but we are in voting for 2.0.1 (RC2) now. If RC2 passes, I'll modified the version to 2.0.2.

            People

            • Assignee:
              yanboliang Yanbo Liang
              Reporter:
              yanboliang Yanbo Liang
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development