Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-17420

Install rmarkdown R package on Jenkins machines

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • None
    • Build, SparkR
    • None

    Description

      For building SparkR vignettes on Jenkins machines, we need the rmarkdown R. The package is available at
      https://cran.r-project.org/web/packages/rmarkdown/index.html - I think running something like
      Rscript -e 'install.packages("rmarkdown", repos="http://cran.stat.ucla.edu/")'

      should work

      Attachments

        Issue Links

          Activity

            shivaram Shivaram Venkataraman added a comment - This came up in https://github.com/apache/spark/pull/14980 cc junyangq shaneknapp
            shaneknapp Shane Knapp added a comment -

            on it

            shaneknapp Shane Knapp added a comment - on it
            shaneknapp Shane Knapp added a comment - https://github.com/amplab/amplab-infra/pull/182

            Thanks shaneknapp - It turns out that we also need pandoc (and specifically some recent version of it) to get this to work. Following the instructions on the rmarkdown page https://github.com/rstudio/rmarkdown/blob/master/PANDOC.md#linux I found that there are pre-built packages for CentOS 5 at https://copr.fedorainfracloud.org/coprs/petersen/pandoc-el5/ and CentOS 6 at https://copr.fedorainfracloud.org/coprs/petersen/pandoc/

            Could we get the appropriate version based on what we are running on the AMP cluster ?

            shivaram Shivaram Venkataraman added a comment - Thanks shaneknapp - It turns out that we also need pandoc (and specifically some recent version of it) to get this to work. Following the instructions on the rmarkdown page https://github.com/rstudio/rmarkdown/blob/master/PANDOC.md#linux I found that there are pre-built packages for CentOS 5 at https://copr.fedorainfracloud.org/coprs/petersen/pandoc-el5/ and CentOS 6 at https://copr.fedorainfracloud.org/coprs/petersen/pandoc/ Could we get the appropriate version based on what we are running on the AMP cluster ?

            I guess the way to verify this would be to run `pandoc -v` and we should see 1.17.X.Y as the version. Similarly `pandoc-citeproc -V` should show version 0.9

            shivaram Shivaram Venkataraman added a comment - I guess the way to verify this would be to run `pandoc -v` and we should see 1.17.X.Y as the version. Similarly `pandoc-citeproc -V` should show version 0.9
            shaneknapp Shane Knapp added a comment -

            ok, i got the yum repo disted and it looks like we'll be installing pandoc 1.17.0.3-1.el6

            [root@amp-jenkins-worker-04 ~]# yum list pandoc.x86_64
            Loaded plugins: fastestmirror, security
            Loading mirror speeds from cached hostfile
             * base: mirrors.sonic.net
             * epel: linux.mirrors.es.net
             * extras: repos.lax.quadranet.com
             * rpmforge: mirror.hmc.edu
             * updates: mirrors.sonic.net
            Available Packages
            pandoc.x86_64                                                 1.17.0.3-1.el6                                                 petersen-pandoc
            

            i'll get this installed on the workers now.

            shaneknapp Shane Knapp added a comment - ok, i got the yum repo disted and it looks like we'll be installing pandoc 1.17.0.3-1.el6 [root@amp-jenkins-worker-04 ~]# yum list pandoc.x86_64 Loaded plugins: fastestmirror, security Loading mirror speeds from cached hostfile * base: mirrors.sonic.net * epel: linux.mirrors.es.net * extras: repos.lax.quadranet.com * rpmforge: mirror.hmc.edu * updates: mirrors.sonic.net Available Packages pandoc.x86_64 1.17.0.3-1.el6 petersen-pandoc i'll get this installed on the workers now.
            shaneknapp Shane Knapp added a comment -
            [ root@amp-jenkins-master ] [ ~ ]
            $ pssh -h jenkins_workers.txt -t 0 "yum -q -y install pandoc"
            [1] 10:03:36 [SUCCESS] amp-jenkins-worker-04
            [2] 10:03:48 [SUCCESS] amp-jenkins-worker-05
            [3] 10:03:49 [SUCCESS] amp-jenkins-worker-06
            [4] 10:03:51 [SUCCESS] amp-jenkins-worker-01
            [5] 10:03:52 [SUCCESS] amp-jenkins-worker-03
            [6] 10:03:53 [SUCCESS] amp-jenkins-worker-07
            [7] 10:03:53 [SUCCESS] amp-jenkins-worker-08
            [8] 10:03:55 [SUCCESS] amp-jenkins-worker-02
            

            and just to double check:

            [ root@amp-jenkins-master ] [ ~ ]
            $ pssh -h jenkins_workers.txt -t 0 -i "yum list pandoc | grep x86"
            [1] 10:04:46 [SUCCESS] amp-jenkins-worker-05
            pandoc.x86_64                  1.17.0.3-1.el6                   @petersen-pandoc
            [2] 10:04:46 [SUCCESS] amp-jenkins-worker-04
            pandoc.x86_64                  1.17.0.3-1.el6                   @petersen-pandoc
            [3] 10:04:46 [SUCCESS] amp-jenkins-worker-03
            pandoc.x86_64                  1.17.0.3-1.el6                   @petersen-pandoc
            [4] 10:04:46 [SUCCESS] amp-jenkins-worker-07
            pandoc.x86_64                  1.17.0.3-1.el6                   @petersen-pandoc
            [5] 10:04:46 [SUCCESS] amp-jenkins-worker-06
            pandoc.x86_64                  1.17.0.3-1.el6                   @petersen-pandoc
            [6] 10:04:46 [SUCCESS] amp-jenkins-worker-02
            pandoc.x86_64                  1.17.0.3-1.el6                   @petersen-pandoc
            [7] 10:04:46 [SUCCESS] amp-jenkins-worker-08
            pandoc.x86_64                  1.17.0.3-1.el6                   @petersen-pandoc
            [8] 10:04:46 [SUCCESS] amp-jenkins-worker-01
            pandoc.x86_64                  1.17.0.3-1.el6                   @petersen-pandoc
            
            shaneknapp Shane Knapp added a comment - [ root@amp-jenkins-master ] [ ~ ] $ pssh -h jenkins_workers.txt -t 0 "yum -q -y install pandoc" [1] 10:03:36 [SUCCESS] amp-jenkins-worker-04 [2] 10:03:48 [SUCCESS] amp-jenkins-worker-05 [3] 10:03:49 [SUCCESS] amp-jenkins-worker-06 [4] 10:03:51 [SUCCESS] amp-jenkins-worker-01 [5] 10:03:52 [SUCCESS] amp-jenkins-worker-03 [6] 10:03:53 [SUCCESS] amp-jenkins-worker-07 [7] 10:03:53 [SUCCESS] amp-jenkins-worker-08 [8] 10:03:55 [SUCCESS] amp-jenkins-worker-02 and just to double check: [ root@amp-jenkins-master ] [ ~ ] $ pssh -h jenkins_workers.txt -t 0 -i "yum list pandoc | grep x86" [1] 10:04:46 [SUCCESS] amp-jenkins-worker-05 pandoc.x86_64 1.17.0.3-1.el6 @petersen-pandoc [2] 10:04:46 [SUCCESS] amp-jenkins-worker-04 pandoc.x86_64 1.17.0.3-1.el6 @petersen-pandoc [3] 10:04:46 [SUCCESS] amp-jenkins-worker-03 pandoc.x86_64 1.17.0.3-1.el6 @petersen-pandoc [4] 10:04:46 [SUCCESS] amp-jenkins-worker-07 pandoc.x86_64 1.17.0.3-1.el6 @petersen-pandoc [5] 10:04:46 [SUCCESS] amp-jenkins-worker-06 pandoc.x86_64 1.17.0.3-1.el6 @petersen-pandoc [6] 10:04:46 [SUCCESS] amp-jenkins-worker-02 pandoc.x86_64 1.17.0.3-1.el6 @petersen-pandoc [7] 10:04:46 [SUCCESS] amp-jenkins-worker-08 pandoc.x86_64 1.17.0.3-1.el6 @petersen-pandoc [8] 10:04:46 [SUCCESS] amp-jenkins-worker-01 pandoc.x86_64 1.17.0.3-1.el6 @petersen-pandoc
            shaneknapp Shane Knapp added a comment - https://github.com/amplab/amplab-infra/pull/183
            shaneknapp Shane Knapp added a comment -

            waiting for shivaram's build to finish, then we can merge my tooling updates and close this out.

            shaneknapp Shane Knapp added a comment - waiting for shivaram 's build to finish, then we can merge my tooling updates and close this out.
            shaneknapp Shane Knapp added a comment -

            whoops, forgot citeproc. on it now.

            On Wed, Sep 7, 2016 at 9:53 AM, Shivaram Venkataraman (JIRA)

            shaneknapp Shane Knapp added a comment - whoops, forgot citeproc. on it now. On Wed, Sep 7, 2016 at 9:53 AM, Shivaram Venkataraman (JIRA)
            shaneknapp Shane Knapp added a comment - - edited

            pandoc-citeproc is now also installed:

            $ pssh -h jenkins_workers.txt -i -t 0 "yum -q -y install pandoc-citeproc"
            [1] 10:20:56 [SUCCESS] amp-jenkins-worker-06
            [2] 10:20:56 [SUCCESS] amp-jenkins-worker-03
            [3] 10:20:56 [SUCCESS] amp-jenkins-worker-05
            [4] 10:20:57 [SUCCESS] amp-jenkins-worker-01
            [5] 10:20:57 [SUCCESS] amp-jenkins-worker-04
            [6] 10:20:58 [SUCCESS] amp-jenkins-worker-08
            [7] 10:20:58 [SUCCESS] amp-jenkins-worker-02
            [8] 10:20:58 [SUCCESS] amp-jenkins-worker-07
            

            version checking:

            $ pssh -h jenkins_workers.txt -i -t 0 "pandoc -v|head -1"
            [1] 10:17:18 [SUCCESS] amp-jenkins-worker-02
            pandoc 1.17.0.3
            [2] 10:17:19 [SUCCESS] amp-jenkins-worker-07
            pandoc 1.17.0.3
            [3] 10:17:19 [SUCCESS] amp-jenkins-worker-03
            pandoc 1.17.0.3
            [4] 10:17:19 [SUCCESS] amp-jenkins-worker-05
            pandoc 1.17.0.3
            [5] 10:17:19 [SUCCESS] amp-jenkins-worker-06
            pandoc 1.17.0.3
            [6] 10:17:19 [SUCCESS] amp-jenkins-worker-04
            pandoc 1.17.0.3
            [7] 10:17:19 [SUCCESS] amp-jenkins-worker-08
            pandoc 1.17.0.3
            [8] 10:17:19 [SUCCESS] amp-jenkins-worker-01
            pandoc 1.17.0.3
            

            and:

            $ pssh -h jenkins_workers.txt -i -t 0 "pandoc-citeproc -V"
            [1] 10:21:34 [SUCCESS] amp-jenkins-worker-04
            pandoc-citeproc 0.9.1.1
            [2] 10:21:34 [SUCCESS] amp-jenkins-worker-06
            pandoc-citeproc 0.9.1.1
            [3] 10:21:34 [SUCCESS] amp-jenkins-worker-05
            pandoc-citeproc 0.9.1.1
            [4] 10:21:34 [SUCCESS] amp-jenkins-worker-07
            pandoc-citeproc 0.9.1.1
            [5] 10:21:34 [SUCCESS] amp-jenkins-worker-02
            pandoc-citeproc 0.9.1.1
            [6] 10:21:34 [SUCCESS] amp-jenkins-worker-03
            pandoc-citeproc 0.9.1.1
            [7] 10:21:34 [SUCCESS] amp-jenkins-worker-08
            pandoc-citeproc 0.9.1.1
            [8] 10:21:34 [SUCCESS] amp-jenkins-worker-01
            pandoc-citeproc 0.9.1.1
            
            shaneknapp Shane Knapp added a comment - - edited pandoc-citeproc is now also installed: $ pssh -h jenkins_workers.txt -i -t 0 "yum -q -y install pandoc-citeproc" [1] 10:20:56 [SUCCESS] amp-jenkins-worker-06 [2] 10:20:56 [SUCCESS] amp-jenkins-worker-03 [3] 10:20:56 [SUCCESS] amp-jenkins-worker-05 [4] 10:20:57 [SUCCESS] amp-jenkins-worker-01 [5] 10:20:57 [SUCCESS] amp-jenkins-worker-04 [6] 10:20:58 [SUCCESS] amp-jenkins-worker-08 [7] 10:20:58 [SUCCESS] amp-jenkins-worker-02 [8] 10:20:58 [SUCCESS] amp-jenkins-worker-07 version checking: $ pssh -h jenkins_workers.txt -i -t 0 "pandoc -v|head -1" [1] 10:17:18 [SUCCESS] amp-jenkins-worker-02 pandoc 1.17.0.3 [2] 10:17:19 [SUCCESS] amp-jenkins-worker-07 pandoc 1.17.0.3 [3] 10:17:19 [SUCCESS] amp-jenkins-worker-03 pandoc 1.17.0.3 [4] 10:17:19 [SUCCESS] amp-jenkins-worker-05 pandoc 1.17.0.3 [5] 10:17:19 [SUCCESS] amp-jenkins-worker-06 pandoc 1.17.0.3 [6] 10:17:19 [SUCCESS] amp-jenkins-worker-04 pandoc 1.17.0.3 [7] 10:17:19 [SUCCESS] amp-jenkins-worker-08 pandoc 1.17.0.3 [8] 10:17:19 [SUCCESS] amp-jenkins-worker-01 pandoc 1.17.0.3 and: $ pssh -h jenkins_workers.txt -i -t 0 "pandoc-citeproc -V" [1] 10:21:34 [SUCCESS] amp-jenkins-worker-04 pandoc-citeproc 0.9.1.1 [2] 10:21:34 [SUCCESS] amp-jenkins-worker-06 pandoc-citeproc 0.9.1.1 [3] 10:21:34 [SUCCESS] amp-jenkins-worker-05 pandoc-citeproc 0.9.1.1 [4] 10:21:34 [SUCCESS] amp-jenkins-worker-07 pandoc-citeproc 0.9.1.1 [5] 10:21:34 [SUCCESS] amp-jenkins-worker-02 pandoc-citeproc 0.9.1.1 [6] 10:21:34 [SUCCESS] amp-jenkins-worker-03 pandoc-citeproc 0.9.1.1 [7] 10:21:34 [SUCCESS] amp-jenkins-worker-08 pandoc-citeproc 0.9.1.1 [8] 10:21:34 [SUCCESS] amp-jenkins-worker-01 pandoc-citeproc 0.9.1.1

            shaneknapp The pandoc and pandoc-citeproc errors are gone. The current error in the build is due to a bug in our script. I'm going to mark this issue as resolved. Thanks a lot for your help !

            shivaram Shivaram Venkataraman added a comment - shaneknapp The pandoc and pandoc-citeproc errors are gone. The current error in the build is due to a bug in our script. I'm going to mark this issue as resolved. Thanks a lot for your help !

            People

              shaneknapp Shane Knapp
              shivaram Shivaram Venkataraman
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: