Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.7.0
    • Fix Version/s: 0.7.0
    • Component/s: general
    • Labels:
      None

      Description

      Here's what I see when I try to build it on Ubuntu 12.03 LTS:

      $ export SCALA_HOME=/opt/scala-2.10.2
      $ make spark-deb
      .....
      [INFO] ------------------------------------------------------------------------
      [INFO] BUILD SUCCESS
      [INFO] ------------------------------------------------------------------------
      [INFO] Total time: 15:19:45.772s
      [INFO] Finished at: Thu Sep 19 08:38:33 PDT 2013
      [INFO] Final Memory: 51M/1187M
      [INFO] ------------------------------------------------------------------------
      mkdir -p debian/tmp
      touch build-indep-stamp
      dh_testdir
      dh_testroot
      sh -x debian/install_spark.sh \
      	  --build-dir=`pwd` \
                --doc-dir=/usr/share/doc/spark \
      	  --prefix=debian/spark
      + set -e
      + getopt -n debian/install_spark.sh -o  -l prefix: -l doc-dir: -l lib-dir: -l installed-lib-dir: -l bin-dir: -l source-dir: -l examples-dir: -l build-dir: -- --build-dir=/home/src/bigtop/output/spark/spark-0.8.0 --doc-dir=/usr/share/doc/spark --prefix=debian/spark
      + OPTS= --build-dir '/home/src/bigtop/output/spark/spark-0.8.0' --doc-dir '/usr/share/doc/spark' --prefix 'debian/spark' --
      + [ 0 != 0 ]
      + eval set --  --build-dir '/home/src/bigtop/output/spark/spark-0.8.0' --doc-dir '/usr/share/doc/spark' --prefix 'debian/spark' --
      + set -- --build-dir /home/src/bigtop/output/spark/spark-0.8.0 --doc-dir /usr/share/doc/spark --prefix debian/spark --
      + true
      + BUILD_DIR=/home/src/bigtop/output/spark/spark-0.8.0
      + shift 2
      + true
      + DOC_DIR=/usr/share/doc/spark
      + shift 2
      + true
      + PREFIX=debian/spark
      + shift 2
      + true
      + shift
      + break
      + eval echo $PREFIX
      + echo debian/spark
      + [ -z debian/spark ]
      + eval echo $BUILD_DIR
      + echo /home/src/bigtop/output/spark/spark-0.8.0
      + [ -z /home/src/bigtop/output/spark/spark-0.8.0 ]
      + eval echo $SOURCE_DIR
      + echo
      + [ -z  ]
      + echo Missing param: SOURCE_DIR
      Missing param: SOURCE_DIR
      + usage
      + echo 
      usage: debian/install_spark.sh <options>
        Required not-so-options:
           --build-dir=DIR             path to dist.dir
           --source-dir=DIR            path to package shared files dir
           --prefix=PREFIX             path to install into
      
        Optional options:
           --doc-dir=DIR               path to install docs into [/usr/share/doc/spark]
           --lib-dir=DIR               path to install Spark home [/usr/lib/spark]
           --installed-lib-dir=DIR     path where lib-dir will end up on target system
           --bin-dir=DIR               path to install bins [/usr/bin]
           --examples-dir=DIR          path to install examples [doc-dir/examples]
           ... [ see source for more similar options ]
        
      
      usage: debian/install_spark.sh <options>
        Required not-so-options:
           --build-dir=DIR             path to dist.dir
           --source-dir=DIR            path to package shared files dir
           --prefix=PREFIX             path to install into
      
        Optional options:
           --doc-dir=DIR               path to install docs into [/usr/share/doc/spark]
           --lib-dir=DIR               path to install Spark home [/usr/lib/spark]
           --installed-lib-dir=DIR     path where lib-dir will end up on target system
           --bin-dir=DIR               path to install bins [/usr/bin]
           --examples-dir=DIR          path to install examples [doc-dir/examples]
           ... [ see source for more similar options ]
        
      + exit 1
      

      So it seems that the build part goes fine, but the install one somehow fails.

      1. BIGTOP-1081.patch
        0.6 kB
        Konstantin Boudnik

        Activity

        Hide
        Konstantin Boudnik added a comment -

        install_spark.sh needs to be extended with one more command line parameter. I will patch it shortly.

        Show
        Konstantin Boudnik added a comment - install_spark.sh needs to be extended with one more command line parameter. I will patch it shortly.
        Hide
        Konstantin Boudnik added a comment -

        Also, Spark doesn't support Scala 2.10, hence you need to build with 2.9.3

        Show
        Konstantin Boudnik added a comment - Also, Spark doesn't support Scala 2.10, hence you need to build with 2.9.3
        Hide
        Konstantin Boudnik added a comment - - edited

        How I can pass in the location of the component built under output/ directory? Essentially the whole fix is

        diff --git bigtop-packages/src/deb/spark/rules bigtop-packages/src/deb/spark/rules
        index 8957eb1..91a2c5a 100644
        --- bigtop-packages/src/deb/spark/rules
        +++ bigtop-packages/src/deb/spark/rules
        @@ -48,6 +48,7 @@ install-indep:
                sh -x debian/install_spark.sh \
                  --build-dir=`pwd` \
                   --doc-dir=/usr/share/doc/spark \
        +          --source-dir=blah_blah
                  --prefix=debian/spark
                dh_install -i
                (dh_lintian) || /bin/true
        
        Show
        Konstantin Boudnik added a comment - - edited How I can pass in the location of the component built under output/ directory? Essentially the whole fix is diff --git bigtop-packages/src/deb/spark/rules bigtop-packages/src/deb/spark/rules index 8957eb1..91a2c5a 100644 --- bigtop-packages/src/deb/spark/rules +++ bigtop-packages/src/deb/spark/rules @@ -48,6 +48,7 @@ install-indep: sh -x debian/install_spark.sh \ --build-dir=`pwd` \ --doc-dir=/usr/share/doc/spark \ + --source-dir=blah_blah --prefix=debian/spark dh_install -i (dh_lintian) || /bin/true
        Hide
        Roman Shaposhnik added a comment -

        Konstantin Boudnik Update: I've enabled Spark builds for the upcoming Bigtop 0.7.0 and it seems that the problem isn't just limited to DEB side of things. Could you please take a look? http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Spark/

        Show
        Roman Shaposhnik added a comment - Konstantin Boudnik Update: I've enabled Spark builds for the upcoming Bigtop 0.7.0 and it seems that the problem isn't just limited to DEB side of things. Could you please take a look? http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Spark/
        Hide
        Roman Shaposhnik added a comment -

        Konstantin Boudnik Do you need to pass the location of the auxiliary files (the ones that come from bigtop-packages/src/common/spark/) or the actual location of the build itself?

        Show
        Roman Shaposhnik added a comment - Konstantin Boudnik Do you need to pass the location of the auxiliary files (the ones that come from bigtop-packages/src/common/spark/) or the actual location of the build itself?
        Hide
        Konstantin Boudnik added a comment - - edited

        Update: I've enabled Spark builds for the upcoming Bigtop 0.7.0 and it seems that the problem isn't just limited to DEB side of things

        this is a different issue: it seems that assembly files aren't found under

        /mnt/jenkins/workspace/Bigtop-trunk-Spark/label/centos5/build/spark/rpm/BUILD/spark-branch-0.8/assembly/target/spark-assembly-*-dist.tar.gz

        It seems that the version of the artifact got changes and the mask

        -*-

        doesn't work anymore. It needs to be

        _*-

        (new file name is spark-assembly_2.9.3-0.8.0-incubating-SNAPSHOT-dist.tar.gz)

        Do you need to pass the location of the auxiliary files?

        for --source-dir I need bigtop-packages/src/common/spark. Now sure how to refer to it.

        There seems to be another issue. For RPM it runs:
        sh /mnt/jenkins/workspace/Bigtop-trunk-Spark/label/centos5/build/spark//rpm//SOURCES/install_spark.sh --build-dir=/mnt/jenkins/workspace/Bigtop-trunk-Spark/label/centos5/build/spark/rpm/BUILD/spark-branch-0.8 --source-dir=/mnt/jenkins/workspace/Bigtop-trunk-Spark/label/centos5/build/spark//rpm//SOURCES --prefix=/var/tmp/spark-0.8.0.2-1-L31331 --doc-dir=/usr/share/doc/spark-0.8.0.2

        But it looks that deb's build layout differs from that of rpm and the results of mvn build are place under output/spark instead of build/spark. Am I missing something?

        Show
        Konstantin Boudnik added a comment - - edited Update: I've enabled Spark builds for the upcoming Bigtop 0.7.0 and it seems that the problem isn't just limited to DEB side of things this is a different issue: it seems that assembly files aren't found under /mnt/jenkins/workspace/Bigtop-trunk-Spark/label/centos5/build/spark/rpm/BUILD/spark-branch-0.8/assembly/target/spark-assembly-*-dist.tar.gz It seems that the version of the artifact got changes and the mask -*- doesn't work anymore. It needs to be _*- (new file name is spark-assembly_2.9.3-0.8.0-incubating-SNAPSHOT-dist.tar.gz ) Do you need to pass the location of the auxiliary files? for --source-dir I need bigtop-packages/src/common/spark . Now sure how to refer to it. There seems to be another issue. For RPM it runs: sh /mnt/jenkins/workspace/Bigtop-trunk-Spark/label/centos5/build/spark//rpm//SOURCES/install_spark.sh --build-dir=/mnt/jenkins/workspace/Bigtop-trunk-Spark/label/centos5/build/spark/rpm/BUILD/spark-branch-0.8 --source-dir=/mnt/jenkins/workspace/Bigtop-trunk-Spark/label/centos5/build/spark//rpm//SOURCES --prefix=/var/tmp/spark-0.8.0.2-1-L31331 --doc-dir=/usr/share/doc/spark-0.8.0.2 But it looks that deb's build layout differs from that of rpm and the results of mvn build are place under output/spark instead of build/spark . Am I missing something?
        Hide
        Roman Shaposhnik added a comment -

        I see what you mean. In that case, I think that the safest thing to do on debian would actually be:

        --source-dir=${PWD}
        

        The reason this is convoluted is because we try not to depend on the source bits that the component's build system itself doesn't make part of the binary assembly.

        IOW, everything that's not part of --build-dir is considered to be off-limits most of the time.

        Show
        Roman Shaposhnik added a comment - I see what you mean. In that case, I think that the safest thing to do on debian would actually be: --source-dir=${PWD} The reason this is convoluted is because we try not to depend on the source bits that the component's build system itself doesn't make part of the binary assembly. IOW, everything that's not part of --build-dir is considered to be off-limits most of the time.
        Hide
        Konstantin Boudnik added a comment -

        Sure, however there some shortcuts that I have to make because Spark original scripts aren't fully prepped to be useful without source workspace in place. Hence, I carry my own. Yeah, yeah - I know: this is quite stupid.

        Show
        Konstantin Boudnik added a comment - Sure, however there some shortcuts that I have to make because Spark original scripts aren't fully prepped to be useful without source workspace in place. Hence, I carry my own. Yeah, yeah - I know: this is quite stupid.
        Hide
        Konstantin Boudnik added a comment -

        that seems to work properly.

        Show
        Konstantin Boudnik added a comment - that seems to work properly.
        Hide
        Roman Shaposhnik added a comment -

        +1.

        Show
        Roman Shaposhnik added a comment - +1.
        Hide
        Konstantin Boudnik added a comment -

        Fixed on master as ca519e0

        Show
        Konstantin Boudnik added a comment - Fixed on master as ca519e0

          People

          • Assignee:
            Konstantin Boudnik
            Reporter:
            Roman Shaposhnik
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development