Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0
    • Fix Version/s: 1.0.0
    • Component/s: build, spark
    • Labels:
      None

      Description

      Update rather old spark dependency.

      This update is not needed by me, but I had to test it for BIGTOP-1549 anyway and it compiled cleanly.

      I cannot test it, though

      1. 0001-BIGTOP-1550.-update-Spark-to-1.1.0.patch
        0.7 kB
        Olaf Flebbe
      2. BIGTOP-1550_sparksubmit.patch
        2 kB
        jay vyas
      3. BIGTOP-1550.patch
        3 kB
        jay vyas

        Activity

        Hide
        jayunit100 jay vyas added a comment -

        Thanks Olaf!!!

        My wife is gonna hate this but I will review this patch today ....

        It's definetely overdue to update our spark to 1x

        Show
        jayunit100 jay vyas added a comment - Thanks Olaf!!! My wife is gonna hate this but I will review this patch today .... It's definetely overdue to update our spark to 1x
        Hide
        jayunit100 jay vyas added a comment -

        (update on this - build failed for me but its my machine , something w/ rpm build. gonna look into it now . probably mvn related, I'm hacking on an old laptop.., )

        Show
        jayunit100 jay vyas added a comment - (update on this - build failed for me but its my machine , something w/ rpm build. gonna look into it now . probably mvn related, I'm hacking on an old laptop.., )
        Hide
        rvs Roman Shaposhnik added a comment -

        jay vyas any chance you can get to the bottom of this (not that Thanksgiving break is over ). If so – perhaps assigning JIRA to you would make sense in the interim.

        Show
        rvs Roman Shaposhnik added a comment - jay vyas any chance you can get to the bottom of this (not that Thanksgiving break is over ). If so – perhaps assigning JIRA to you would make sense in the interim.
        Hide
        jayunit100 jay vyas added a comment -

        hey ! playing w it now will let you know

        Show
        jayunit100 jay vyas added a comment - hey ! playing w it now will let you know
        Hide
        jayunit100 jay vyas added a comment - - edited

        Overall, it looks like the patch does the right things, but i got some minor errors
        i will look more into it, and feedback from others is welcome. sure, feel free to assign it to me !

        Show
        jayunit100 jay vyas added a comment - - edited Overall, it looks like the patch does the right things, but i got some minor errors i will look more into it, and feedback from others is welcome. sure, feel free to assign it to me !
        Hide
        jayunit100 jay vyas added a comment -

        1) mostly build just fine sorry, i was having some stochastic, unreproducible errors (internet or memory related).

        2) However after successfull build, I noticed there is no spark-submit found using rpm -qlp. So that should be fixed.

        For those interested here are the outputs.

         
        rpm -qlp ./output/spark/noarch/*rpm | grep bin
        /usr/bin/spark-executor
        /usr/bin/spark-shell
        /usr/lib/spark/bin
        /usr/lib/spark/bin/compute-classpath.sh
        /usr/lib/spark/bin/load-spark-env.sh
        /usr/lib/spark/bin/spark-class
        /usr/lib/spark/bin/spark-shell
        /usr/lib/spark/bin/utils.sh
        /usr/lib/spark/sbin
        /usr/lib/spark/sbin/slaves.sh
        /usr/lib/spark/sbin/spark-config.sh
        /usr/lib/spark/sbin/spark-daemon.sh
        /usr/lib/spark/sbin/spark-daemons.sh
        /usr/lib/spark/sbin/spark-executor
        /usr/lib/spark/sbin/start-all.sh
        /usr/lib/spark/sbin/start-history-server.sh
        /usr/lib/spark/sbin/start-master.sh
        /usr/lib/spark/sbin/start-slave.sh
        /usr/lib/spark/sbin/start-slaves.sh
        /usr/lib/spark/sbin/start-thriftserver.sh
        /usr/lib/spark/sbin/stop-all.sh
        /usr/lib/spark/sbin/stop-history-server.sh
        /usr/lib/spark/sbin/stop-master.sh
        /usr/lib/spark/sbin/stop-slaves.sh
        /usr/bin/pyspark
        /usr/lib/spark/bin/pyspark
        

        thanks Olaf Flebbe .,... quite close to an updated spark on bigtop !

        Show
        jayunit100 jay vyas added a comment - 1) mostly build just fine sorry, i was having some stochastic, unreproducible errors (internet or memory related). 2) However after successfull build, I noticed there is no spark-submit found using rpm -qlp. So that should be fixed. For those interested here are the outputs. rpm -qlp ./output/spark/noarch/*rpm | grep bin /usr/bin/spark-executor /usr/bin/spark-shell /usr/lib/spark/bin /usr/lib/spark/bin/compute-classpath.sh /usr/lib/spark/bin/load-spark-env.sh /usr/lib/spark/bin/spark-class /usr/lib/spark/bin/spark-shell /usr/lib/spark/bin/utils.sh /usr/lib/spark/sbin /usr/lib/spark/sbin/slaves.sh /usr/lib/spark/sbin/spark-config.sh /usr/lib/spark/sbin/spark-daemon.sh /usr/lib/spark/sbin/spark-daemons.sh /usr/lib/spark/sbin/spark-executor /usr/lib/spark/sbin/start-all.sh /usr/lib/spark/sbin/start-history-server.sh /usr/lib/spark/sbin/start-master.sh /usr/lib/spark/sbin/start-slave.sh /usr/lib/spark/sbin/start-slaves.sh /usr/lib/spark/sbin/start-thriftserver.sh /usr/lib/spark/sbin/stop-all.sh /usr/lib/spark/sbin/stop-history-server.sh /usr/lib/spark/sbin/stop-master.sh /usr/lib/spark/sbin/stop-slaves.sh /usr/bin/pyspark /usr/lib/spark/bin/pyspark thanks Olaf Flebbe .,... quite close to an updated spark on bigtop !
        Hide
        jayunit100 jay vyas added a comment - - edited

        okay !

        • heres a fix for spark-submit.
        • combining this w/ olaf's patch should get us a basic working patch for packaging spark 1.1.0..

        Next step will be to test that the RPM built works etc etc... But applying these two patches seems to get us up to 1.1.0 and builds clean!

        Show
        jayunit100 jay vyas added a comment - - edited okay ! heres a fix for spark-submit. combining this w/ olaf's patch should get us a basic working patch for packaging spark 1.1.0.. Next step will be to test that the RPM built works etc etc... But applying these two patches seems to get us up to 1.1.0 and builds clean!
        Hide
        rvs Roman Shaposhnik added a comment -

        jay vyas ping! any progress on this one?

        Show
        rvs Roman Shaposhnik added a comment - jay vyas ping! any progress on this one?
        Hide
        jayunit100 jay vyas added a comment -

        yup ! as per the patch i attached, if you combine that with Olaf's, we now
        can build spark 1.1 , and package spark-submit into the rpm, which is also necessary.

        Just two things to be done, after applying the above 2 patches.
        (1) Need to add the equivalent of my patch's updates into deb as well
        (2) need to run a smoke test to make sure spark still works.

        otherwise, i think its pretty close ! ill look into it more this wk

        Show
        jayunit100 jay vyas added a comment - yup ! as per the patch i attached, if you combine that with Olaf's, we now can build spark 1.1 , and package spark-submit into the rpm, which is also necessary. Just two things to be done, after applying the above 2 patches. (1) Need to add the equivalent of my patch's updates into deb as well (2) need to run a smoke test to make sure spark still works. otherwise, i think its pretty close ! ill look into it more this wk
        Hide
        evans_ye Evans Ye added a comment -

        Generally the patch looks good. I can successfully build up spark 1.1 RPM packages including spark-submit.
        I'll have some more test on it, like running a spark cluster.

        Show
        evans_ye Evans Ye added a comment - Generally the patch looks good. I can successfully build up spark 1.1 RPM packages including spark-submit. I'll have some more test on it, like running a spark cluster.
        Hide
        evans_ye Evans Ye added a comment -

        I got my testing cluster installed the Spark 1.1 RPMs I just built.
        The system and daemons seems functional, and a testing job I submitted had success as well.
        In summary, +1 on the patch.
        BTW jay vyas you will work on the debian stuff in this patch, too?

        Show
        evans_ye Evans Ye added a comment - I got my testing cluster installed the Spark 1.1 RPMs I just built. The system and daemons seems functional, and a testing job I submitted had success as well. In summary, +1 on the patch. BTW jay vyas you will work on the debian stuff in this patch, too?
        Hide
        jayunit100 jay vyas added a comment - - edited

        i didnt get debian in. i can look to see how to do that ....
        thanks for the review !

        i better get debian working and will clean up the patch and resubmit

        Show
        jayunit100 jay vyas added a comment - - edited i didnt get debian in. i can look to see how to do that .... thanks for the review ! i better get debian working and will clean up the patch and resubmit
        Hide
        jayunit100 jay vyas added a comment - - edited

        okay, heres a clean patch to try out ! anyone want to test on deb ? either way i think its a good step forward and we can probably go with it ......

        ill wait for the next review of the patch as a whole. possibly some of our guys who have deb setups maybe Sean Mackrory or Konstantin Boudnik or Roman Shaposhnik will like to give it a whilrl.

        Show
        jayunit100 jay vyas added a comment - - edited okay, heres a clean patch to try out ! anyone want to test on deb ? either way i think its a good step forward and we can probably go with it ...... ill wait for the next review of the patch as a whole. possibly some of our guys who have deb setups maybe Sean Mackrory or Konstantin Boudnik or Roman Shaposhnik will like to give it a whilrl.
        Hide
        cos Konstantin Boudnik added a comment -

        jay vyas, should you reassign this JIRA to yourself if you're working on it?

        Show
        cos Konstantin Boudnik added a comment - jay vyas , should you reassign this JIRA to yourself if you're working on it?
        Hide
        rvs Roman Shaposhnik added a comment -

        jay vyas please let us know if DEB related help is required. I am sure that between Olaf Flebbe and myself we can get this to the finish line wrt. Debian functionality.

        Show
        rvs Roman Shaposhnik added a comment - jay vyas please let us know if DEB related help is required. I am sure that between Olaf Flebbe and myself we can get this to the finish line wrt. Debian functionality.
        Hide
        jayunit100 jay vyas added a comment -

        @rvs i think its ready for external review now . feel free to +1 / -1 if you think its ready for commit !

        Show
        jayunit100 jay vyas added a comment - @rvs i think its ready for external review now . feel free to +1 / -1 if you think its ready for commit !
        Hide
        oflebbe Olaf Flebbe added a comment -

        Debian compiles and the spark-core package contains spark-submit .

        dpkg-deb -c spark-core_1.1.0-1_all.deb| grep submit
        -rwxr-xr-x root/root       142 2014-12-15 09:56 ./usr/bin/spark-submit
        

        Since I had no time until now, to dig into smoke testing on Debian at all, add a +1 for me.

        Show
        oflebbe Olaf Flebbe added a comment - Debian compiles and the spark-core package contains spark-submit . dpkg-deb -c spark-core_1.1.0-1_all.deb| grep submit -rwxr-xr-x root/root 142 2014-12-15 09:56 ./usr/bin/spark-submit Since I had no time until now, to dig into smoke testing on Debian at all, add a +1 for me.
        Hide
        rvs Roman Shaposhnik added a comment -

        +1 from my side. jay vyas can you please commit?

        Show
        rvs Roman Shaposhnik added a comment - +1 from my side. jay vyas can you please commit?
        Hide
        jayunit100 jay vyas added a comment -

        commiteed Roman Shaposhnik ! thanks Olaf Flebbe olaf thanks alot for initially attaching the change to the spark version that got our butts in gear ...

        Show
        jayunit100 jay vyas added a comment - commiteed Roman Shaposhnik ! thanks Olaf Flebbe olaf thanks alot for initially attaching the change to the spark version that got our butts in gear ...

          People

          • Assignee:
            jayunit100 jay vyas
            Reporter:
            oflebbe Olaf Flebbe
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development