Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-12910

Support for specifying version of R to use while creating sparkR libraries

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments


    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • 2.0.0
    • SparkR
    • Linux

    • Patch


      When we use `$SPARK_HOME/R/install-dev.sh` it uses the default system R. However, a user might have locally installed their own version of R. There should be a way to specify which R version to use.

      I have fixed this in my code using the following patch:

      $ git diff HEAD
      diff --git a/R/README.md b/R/README.md
      index 005f56d..99182e5 100644
      --- a/R/README.md
      +++ b/R/README.md
      @@ -1,6 +1,15 @@
       # R on Spark
       SparkR is an R package that provides a light-weight frontend to use Spark from R.
      +### Installing sparkR
      +Libraries of sparkR need to be created in `$SPARK_HOME/R/lib`. This can be done by running the script `$SPARK_HOME/R/install-dev.sh`.
      +By default the above script uses the system wide installation of R. However, this can be changed to any user installed location of R by giving the full path of the `$R_HOME` as the first argument to the install-dev.sh script.
      +# where /home/username/R is where R is installed and /home/username/R/bin contains the files R and RScript
      +./install-dev.sh /home/username/R 
       ### SparkR development
      diff --git a/R/install-dev.sh b/R/install-dev.sh
      index 4972bb9..a8efa86 100755
      --- a/R/install-dev.sh
      +++ b/R/install-dev.sh
      @@ -35,12 +35,19 @@ LIB_DIR="$FWDIR/lib"
       mkdir -p $LIB_DIR
       pushd $FWDIR > /dev/null
      +if [ ! -z "$1" ]
      +  then
      +    R_HOME="$1/bin"
      +   else
      +    R_HOME="$(dirname $(which R))"
      +echo "USING R_HOME = $R_HOME"
       # Generate Rd files if devtools is installed
      -Rscript -e ' if("devtools" %in% rownames(installed.packages())) { library(devtools); devtools::document(pkg="./pkg", roclets=c("rd")) }'
      +"$R_HOME/"Rscript -e ' if("devtools" %in% rownames(installed.packages())) { library(devtools); devtools::document(pkg="./pkg", roclets=c("rd")) }'
       # Install SparkR to $LIB_DIR
      -R CMD INSTALL --library=$LIB_DIR $FWDIR/pkg/
      +"$R_HOME/"R CMD INSTALL --library=$LIB_DIR $FWDIR/pkg/
       # Zip the SparkR package so that it can be distributed to worker nodes on YARN
       cd $LIB_DIR



          This comment will be Viewable by All Users Viewable by All Users


            shubhanshumishra@gmail.com Shubhanshu Mishra
            shubhanshumishra@gmail.com Shubhanshu Mishra
            0 Vote for this issue
            3 Start watching this issue




                Issue deployment