Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 1.0.0
    • Component/s: build, debian, rpm
    • Labels:

      Description

      Lets package (amplab) tachyon into bigtop. Its synergistic for both mapreduce, yarn, and spark , which are the dominant processing engines in big data.

      1. bigtop-1497-all.patch
        41 kB
        Huamin Chen
      2. bigtop-1497-rebase.patch
        37 kB
        Huamin Chen

        Issue Links

          Activity

          Hide
          jayunit100 jay vyas added a comment -

          I will try to learn some more about packaging and play some with this . Meanwhile, for those interested - here are the some starter points around packaging stuff for bigtop.

          0) top level build.gradle file defines all bigtop tasks. it sources a packages.gradle file that has the packaging tasks.

          1) The bigtop.mk file defines all components in bigtop. it is referenced by packages.gradle.

          2) Each component has a do-component-build script. That script builds the java project for you.

          3) For deb and rpm specifics, those builders are in respective bigtop-packages/ directories. They
          each use do-component-build to get the core jars. So the first step is to write do-component-build for tachyon, i think.

          4) For smoke tests, you can just dump a groovy script into bigtop-tests/smoke-tests/tachyon.

          For a simple example of how to package a single jar app, like mahout, you can see the
          blog post : https://blogs.apache.org/bigtop/entry/getting_involved_with_bigtop_packaging .

          Show
          jayunit100 jay vyas added a comment - I will try to learn some more about packaging and play some with this . Meanwhile, for those interested - here are the some starter points around packaging stuff for bigtop. 0) top level build.gradle file defines all bigtop tasks. it sources a packages.gradle file that has the packaging tasks. 1) The bigtop.mk file defines all components in bigtop. it is referenced by packages.gradle. 2) Each component has a do-component-build script. That script builds the java project for you. 3) For deb and rpm specifics, those builders are in respective bigtop-packages/ directories. They each use do-component-build to get the core jars. So the first step is to write do-component-build for tachyon, i think. 4) For smoke tests, you can just dump a groovy script into bigtop-tests/smoke-tests/tachyon. For a simple example of how to package a single jar app, like mahout, you can see the blog post : https://blogs.apache.org/bigtop/entry/getting_involved_with_bigtop_packaging .
          Hide
          jayunit100 jay vyas added a comment -

          Sean Mackrory can you help us w/ the deb packaging if we get the RPM packaging working ?

          Show
          jayunit100 jay vyas added a comment - Sean Mackrory can you help us w/ the deb packaging if we get the RPM packaging working ?
          Hide
          tstclair Timothy St. Clair added a comment -

          You could likely convert http://pkgs.fedoraproject.org/cgit/amplab-tachyon.git/tree/ to support el7.

          The instructions would be the same as outlined here:
          http://timothysc.github.io/blog/2014/02/17/bdas-tachyon/

          Show
          tstclair Timothy St. Clair added a comment - You could likely convert http://pkgs.fedoraproject.org/cgit/amplab-tachyon.git/tree/ to support el7. The instructions would be the same as outlined here: http://timothysc.github.io/blog/2014/02/17/bdas-tachyon/
          Hide
          jayunit100 jay vyas added a comment -

          Thanks Tim. Great that fedora already has tachyon packaged. Will definetly be a big help for us....

          Show
          jayunit100 jay vyas added a comment - Thanks Tim. Great that fedora already has tachyon packaged. Will definetly be a big help for us....
          Hide
          mackrorysd Sean Mackrory added a comment - - edited

          jay vyas Yeah I'd be happy to. Also, I'd suggest your list is missing something that's just as important as do-component-build:

          5) Each component has an install_<component>.sh script. This is responsible for copying the files from the compiled tarball into the layout that should exist in the filesystem when the package is installed (at least as far as is common between distros).

          Show
          mackrorysd Sean Mackrory added a comment - - edited jay vyas Yeah I'd be happy to. Also, I'd suggest your list is missing something that's just as important as do-component-build: 5) Each component has an install_<component>.sh script. This is responsible for copying the files from the compiled tarball into the layout that should exist in the filesystem when the package is installed (at least as far as is common between distros).
          Hide
          jayunit100 jay vyas added a comment - - edited

          thanks for that correction sean ... in any case Roman Shaposhnik I beleive you are indeed taking this patch over? Sean Mackrory maybe you can help me to flesh out https://cwiki.apache.org/confluence/display/BIGTOP/Packaging+Cookbook. ? I just created that to formalize the stuff you said above .................

          Show
          jayunit100 jay vyas added a comment - - edited thanks for that correction sean ... in any case Roman Shaposhnik I beleive you are indeed taking this patch over? Sean Mackrory maybe you can help me to flesh out https://cwiki.apache.org/confluence/display/BIGTOP/Packaging+Cookbook . ? I just created that to formalize the stuff you said above .................
          Hide
          rvs Roman Shaposhnik added a comment -

          jay vyas indeed I was planning to. Ran out of cycles in a pre-vacation rush Any chance you guys can pick this one while I'm gone (for 3 weeks)?

          Show
          rvs Roman Shaposhnik added a comment - jay vyas indeed I was planning to. Ran out of cycles in a pre-vacation rush Any chance you guys can pick this one while I'm gone (for 3 weeks)?
          Hide
          jayunit100 jay vyas added a comment -

          sounds good. I will look into this during the meantime !

          Show
          jayunit100 jay vyas added a comment - sounds good. I will look into this during the meantime !
          Hide
          jayunit100 jay vyas added a comment - - edited

          Sean Mackrory

          Heres our first stab at tachyon packaging. How does this look to you ?
          https://github.com/rootfs/bigtop/compare/tachyon?expand=1

          We can add the deb stuff after we get the RPM side of things right i think.

          cc Huamin Chen

          Show
          jayunit100 jay vyas added a comment - - edited Sean Mackrory Heres our first stab at tachyon packaging. How does this look to you ? https://github.com/rootfs/bigtop/compare/tachyon?expand=1 We can add the deb stuff after we get the RPM side of things right i think. cc Huamin Chen
          Hide
          mackrorysd Sean Mackrory added a comment -

          Looks good so far. A few things that jump out are

          1) The install section where directories are created and populated with files from the build should be abstracted into an install_tachyon.sh script - like do-component-build we implement that for each component so as to minimize the stuff in platform-specific files like .spec files.
          2) The .spec file has a reference directly to Amp Lab's Github for the tarball. Information about where to obtain tarball releases should be in the bigtop.mk file, and package.mk or the Gradle will download the tarball so that neither the .spec file or the Debian packaging has to know specific about where it's coming from.
          3) This might be a case where Bigtop has deviated from what the Linux distributions do, but most of the bits like JARs, shell scripts from upstream, etc. go under /usr/lib/<component>/, and /usr/bin contains wrappers that set up the environment, detect Java, etc. and then invoke scripts in/usr/lib/<component>/bin.

          Are there init scripts, etc. for the services that make up this system? We have a template (look for uses of init.d.tmpl) for generating init scripts that are aware of the little quicks on various distros.

          Show
          mackrorysd Sean Mackrory added a comment - Looks good so far. A few things that jump out are 1) The install section where directories are created and populated with files from the build should be abstracted into an install_tachyon.sh script - like do-component-build we implement that for each component so as to minimize the stuff in platform-specific files like .spec files. 2) The .spec file has a reference directly to Amp Lab's Github for the tarball. Information about where to obtain tarball releases should be in the bigtop.mk file, and package.mk or the Gradle will download the tarball so that neither the .spec file or the Debian packaging has to know specific about where it's coming from. 3) This might be a case where Bigtop has deviated from what the Linux distributions do, but most of the bits like JARs, shell scripts from upstream, etc. go under /usr/lib/<component>/, and /usr/bin contains wrappers that set up the environment, detect Java, etc. and then invoke scripts in/usr/lib/<component>/bin. Are there init scripts, etc. for the services that make up this system? We have a template (look for uses of init.d.tmpl) for generating init scripts that are aware of the little quicks on various distros.
          Hide
          chenh Huamin Chen added a comment -

          thank Sean Mackrory, I'll fix them. Tachyon doesn't ship init scripts/services. But in Fedora, we added init scripts for Tachyon. What do you think if I port these scripts to bigtop?

          Show
          chenh Huamin Chen added a comment - thank Sean Mackrory , I'll fix them. Tachyon doesn't ship init scripts/services. But in Fedora, we added init scripts for Tachyon. What do you think if I port these scripts to bigtop?
          Hide
          mackrorysd Sean Mackrory added a comment -

          I believe all of the init scripts in Bigtop packages are maintained by Bigtop and not the upstream projects. I'd recommend you use the template instead because it takes care of several differences between Red Hat-based distros and Debian, and in general just follows all the conventions that the rest of Bigtop does.

          Show
          mackrorysd Sean Mackrory added a comment - I believe all of the init scripts in Bigtop packages are maintained by Bigtop and not the upstream projects. I'd recommend you use the template instead because it takes care of several differences between Red Hat-based distros and Debian, and in general just follows all the conventions that the rest of Bigtop does.
          Hide
          jayunit100 jay vyas added a comment - - edited

          Hi Huamin Chen . i pulled from your fork but don't see the jar being built anywhere ?

          [jay@rhbd bigtop]$ rpm -qpl build/tachyon/rpm/RPMS/noarch/tachyon-0.5.0-1.fc20.noarch.rpm
          /etc/tachyon
          /etc/tachyon/log4j.properties
          /etc/tachyon/slaves
          /usr/bin/tachyon
          /usr/bin/tachyon-mount.sh
          /usr/bin/tachyon-slaves.sh
          /usr/bin/tachyon-start.sh
          /usr/bin/tachyon-stop.sh
          /usr/libexec/tachyon-config.sh
          /usr/libexec/tachyon-layout.sh.linux.template
          /usr/share/doc/tachyon
          /usr/share/doc/tachyon/LICENSE
          /usr/share/doc/tachyon/README.md
          /usr/share/tachyon
          /usr/share/tachyon/web
          /usr/share/tachyon/web/webapp
          /usr/share/tachyon/web/webapp/browse.jsp
          /usr/share/tachyon/web/webapp/configuration.jsp
          /usr/share/tachyon/web/webapp/css
          /usr/share/tachyon/web/webapp/css/bootstrap-responsive.min.css
          /usr/share/tachyon/web/webapp/css/bootstrap.min.css
          /usr/share/tachyon/web/webapp/css/tachyoncustom.min.css
          /usr/share/tachyon/web/webapp/dependency.jsp
          /usr/share/tachyon/web/webapp/general.jsp
          /usr/share/tachyon/web/webapp/img
          /usr/share/tachyon/web/webapp/img/amplab_logo.png
          /usr/share/tachyon/web/webapp/img/glyphicons-halflings-white.png
          /usr/share/tachyon/web/webapp/img/glyphicons-halflings.png
          /usr/share/tachyon/web/webapp/img/logo.png
          /usr/share/tachyon/web/webapp/index.html
          /usr/share/tachyon/web/webapp/js
          /usr/share/tachyon/web/webapp/js/bootstrap.min.js
          /usr/share/tachyon/web/webapp/js/jquery-1.9.1.min.js
          /usr/share/tachyon/web/webapp/memory.jsp
          /usr/share/tachyon/web/webapp/viewFile.jsp
          

          So, looks like we are missing

          [jay@rhbd tmp]$ rpm -qpl ./amplab-tachyon-0.5.0-3.5c054ca.fc21.noarch.rpm  | grep jar
          warning: ./amplab-tachyon-0.5.0-3.5c054ca.fc21.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 95a43f54: NOKEY
          /usr/share/hadoop/common/lib/tachyon-libthrift.jar
          /usr/share/hadoop/common/lib/tachyon.jar
          /usr/share/java/amplab-tachyon/tachyon-client.jar
          /usr/share/java/tachyon.jar
          

          The above jars. I will look more into it.

          Show
          jayunit100 jay vyas added a comment - - edited Hi Huamin Chen . i pulled from your fork but don't see the jar being built anywhere ? [jay@rhbd bigtop]$ rpm -qpl build/tachyon/rpm/RPMS/noarch/tachyon-0.5.0-1.fc20.noarch.rpm /etc/tachyon /etc/tachyon/log4j.properties /etc/tachyon/slaves /usr/bin/tachyon /usr/bin/tachyon-mount.sh /usr/bin/tachyon-slaves.sh /usr/bin/tachyon-start.sh /usr/bin/tachyon-stop.sh /usr/libexec/tachyon-config.sh /usr/libexec/tachyon-layout.sh.linux.template /usr/share/doc/tachyon /usr/share/doc/tachyon/LICENSE /usr/share/doc/tachyon/README.md /usr/share/tachyon /usr/share/tachyon/web /usr/share/tachyon/web/webapp /usr/share/tachyon/web/webapp/browse.jsp /usr/share/tachyon/web/webapp/configuration.jsp /usr/share/tachyon/web/webapp/css /usr/share/tachyon/web/webapp/css/bootstrap-responsive.min.css /usr/share/tachyon/web/webapp/css/bootstrap.min.css /usr/share/tachyon/web/webapp/css/tachyoncustom.min.css /usr/share/tachyon/web/webapp/dependency.jsp /usr/share/tachyon/web/webapp/general.jsp /usr/share/tachyon/web/webapp/img /usr/share/tachyon/web/webapp/img/amplab_logo.png /usr/share/tachyon/web/webapp/img/glyphicons-halflings-white.png /usr/share/tachyon/web/webapp/img/glyphicons-halflings.png /usr/share/tachyon/web/webapp/img/logo.png /usr/share/tachyon/web/webapp/index.html /usr/share/tachyon/web/webapp/js /usr/share/tachyon/web/webapp/js/bootstrap.min.js /usr/share/tachyon/web/webapp/js/jquery-1.9.1.min.js /usr/share/tachyon/web/webapp/memory.jsp /usr/share/tachyon/web/webapp/viewFile.jsp So, looks like we are missing [jay@rhbd tmp]$ rpm -qpl ./amplab-tachyon-0.5.0-3.5c054ca.fc21.noarch.rpm | grep jar warning: ./amplab-tachyon-0.5.0-3.5c054ca.fc21.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 95a43f54: NOKEY /usr/share/hadoop/common/lib/tachyon-libthrift.jar /usr/share/hadoop/common/lib/tachyon.jar /usr/share/java/amplab-tachyon/tachyon-client.jar /usr/share/java/tachyon.jar The above jars. I will look more into it.
          Hide
          chenh Huamin Chen added a comment - - edited

          they are fixed here
          https://github.com/rootfs/bigtop/commit/cb8b7b99066024aad87f4d923638663f9f01bc9c

          jay vyas your repo is not updated, pull again

          Show
          chenh Huamin Chen added a comment - - edited they are fixed here https://github.com/rootfs/bigtop/commit/cb8b7b99066024aad87f4d923638663f9f01bc9c jay vyas your repo is not updated, pull again
          Hide
          jayunit100 jay vyas added a comment -

          ah okay ! great, testing now

          Show
          jayunit100 jay vyas added a comment - ah okay ! great, testing now
          Hide
          jayunit100 jay vyas added a comment - - edited

          looks better now

          /usr/lib/tachyon/libexec/tachyon-layout.sh.linux.template
          /usr/lib/tachyon/tachyon-0.5.0-jar-with-dependencies.jar
          /usr/lib/tachyon/tachyon-0.5.0-tests.jar
          /usr/lib/tachyon/tachyon-0.5.0.jar
          

          tomorrow I will test the installation according to the tachyon docs as well

          Show
          jayunit100 jay vyas added a comment - - edited looks better now /usr/lib/tachyon/libexec/tachyon-layout.sh.linux.template /usr/lib/tachyon/tachyon-0.5.0-jar-with-dependencies.jar /usr/lib/tachyon/tachyon-0.5.0-tests.jar /usr/lib/tachyon/tachyon-0.5.0.jar tomorrow I will test the installation according to the tachyon docs as well
          Hide
          jayunit100 jay vyas added a comment -

          Ilya Tikhonov can you also contrast gridgain w/ tachyon ? we are now thinking about these puppet recipes and how they should work to install everything.

          Show
          jayunit100 jay vyas added a comment - Ilya Tikhonov can you also contrast gridgain w/ tachyon ? we are now thinking about these puppet recipes and how they should work to install everything.
          Hide
          jayunit100 jay vyas added a comment - - edited

          Huamin Chen Hi huamin. okay, making progress. Heres what we need, i can try to push the fixes to the repo:

          • slaves : need a puppet conf to creat that file.
          • tachyon.conf. Not sure who creates this ?
          • (update:IGNORE) tachyon.sh again not sure why its missing, maybe just another file directive
          • tachyon-master.service and tachyon-slave.service files.
          • tachyon-layout.sh Shall we create it in puppet ? i see tachyon.sh.linux.template
          • tachyon-libthrift.jar : Also we need to add this guy
          Show
          jayunit100 jay vyas added a comment - - edited Huamin Chen Hi huamin. okay, making progress. Heres what we need, i can try to push the fixes to the repo: slaves : need a puppet conf to creat that file. tachyon.conf . Not sure who creates this ? (update:IGNORE) tachyon.sh again not sure why its missing, maybe just another file directive tachyon-master.service and tachyon-slave.service files. tachyon-layout.sh Shall we create it in puppet ? i see tachyon.sh.linux.template tachyon-libthrift.jar : Also we need to add this guy
          Hide
          chenh Huamin Chen added a comment -

          slaves is at /etc/tachyon/slaves
          tachyon.conf, .service .sh, tachyon-libthrift.jar are part of Fedora distribution but they don't need to be here.

          Show
          chenh Huamin Chen added a comment - slaves is at /etc/tachyon/slaves tachyon.conf , .service .sh , tachyon-libthrift.jar are part of Fedora distribution but they don't need to be here.
          Hide
          jayunit100 jay vyas added a comment -

          oh okay. So only tachyon.conf.
          Should we use puppet to create that?

          Show
          jayunit100 jay vyas added a comment - oh okay. So only tachyon.conf . Should we use puppet to create that?
          Hide
          jayunit100 jay vyas added a comment -

          FYI, testing this in the docker containers now according to standard tachyon deploy instructions

          Show
          jayunit100 jay vyas added a comment - FYI, testing this in the docker containers now according to standard tachyon deploy instructions
          Hide
          jayunit100 jay vyas added a comment - - edited
          • ./bigtop-packages/src/common/tachyon/install_tachyon.sh seems to be bundling my local build path into the shell script.

          otherwise looking pretty good. hacking on it some more to see ....

          • Also, running tachyon tasks, i get class not found errors. should the shell script know how to add /usr/lib/tachyon.... jars to the classpath ?
          Show
          jayunit100 jay vyas added a comment - - edited ./bigtop-packages/src/common/tachyon/install_tachyon.sh seems to be bundling my local build path into the shell script. otherwise looking pretty good. hacking on it some more to see .... Also, running tachyon tasks, i get class not found errors. should the shell script know how to add /usr/lib/tachyon.... jars to the classpath ?
          Hide
          jayunit100 jay vyas added a comment -

          Okay ! Fixxed some java_home issues in the rootfs/bigtop (branch = tachyon) any one wanting totest can
          pull and test from there.

          I will test actual storage of tachyon data tomorrow. It is now correctly calling the tachyon binaries etc.

          https://github.com/rootfs/bigtop

          Show
          jayunit100 jay vyas added a comment - Okay ! Fixxed some java_home issues in the rootfs/bigtop (branch = tachyon) any one wanting totest can pull and test from there. I will test actual storage of tachyon data tomorrow. It is now correctly calling the tachyon binaries etc. https://github.com/rootfs/bigtop
          Hide
          hsaputra Henry Saputra added a comment -

          W00t!

          Show
          hsaputra Henry Saputra added a comment - W00t!
          Hide
          jayunit100 jay vyas added a comment - - edited

          Okay, tachyon slaves/master install just fine now

          1) rpm -i /bigtop-home/build/tachyon/rpm/RPMS/noarch/tachyon-0.5.0-1.fc20.noarch.rpm

          2) /usr/bin/tachyon format

          3) /usr/lib/tachyon/bin/tachyon-start.sh all Mount

          Then jps shows:

          16066 SecondaryNameNode
          1082 TachyonWorker
          22486 NodeManager
          16859 DataNode
          883 TachyonMaster
          25323 -- process information unavailable
          16396 WebAppProxyServer
          16608 NameNode
          1122 Jps
          22071 ResourceManager
          22774 JobHistoryServer
          
          Show
          jayunit100 jay vyas added a comment - - edited Okay, tachyon slaves/master install just fine now 1) rpm -i /bigtop-home/build/tachyon/rpm/RPMS/noarch/tachyon-0.5.0-1.fc20.noarch.rpm 2) /usr/bin/tachyon format 3) /usr/lib/tachyon/bin/tachyon-start.sh all Mount Then jps shows: 16066 SecondaryNameNode 1082 TachyonWorker 22486 NodeManager 16859 DataNode 883 TachyonMaster 25323 -- process information unavailable 16396 WebAppProxyServer 16608 NameNode 1122 Jps 22071 ResourceManager 22774 JobHistoryServer
          Hide
          jayunit100 jay vyas added a comment -

          And in isolation, tachyon is definetly working.

          [root@bigtop1 ~]# tachyon runTest Basic MUST_CACHE
          /BasicFile_MUST_CACHE has been removed
          2014-11-14 23:48:33,188 INFO   (TachyonFS.java:connect) - Trying to connect master @ localhost/127.0.0.1:19998
          2014-11-14 23:48:33,242 INFO   (MasterClient.java:getUserId) - User registered at the master localhost/127.0.0.1:19998 got UserId 7
          2014-11-14 23:48:33,243 INFO   (TachyonFS.java:connect) - Trying to get local worker host : bigtop1.vagrant
          2014-11-14 23:48:33,254 INFO   (TachyonFS.java:connect) - Connecting local worker @ bigtop1.vagrant/10.10.10.11:29998
          2014-11-14 23:48:33,282 INFO   (CommonUtils.java:printTimeTakenMs) - createFile with fileId 4 took 96 ms.
          2014-11-14 23:48:33,320 INFO   (TachyonFS.java:createAndGetUserTempFolder) - Folder /mnt/ramdisk/tachyonworker/users/7 was created!
          2014-11-14 23:48:33,331 INFO   (BlockOutStream.java:<init>) - /mnt/ramdisk/tachyonworker/users/7/4294967296 was created!
          Passed the test!
          
          

          Sean Mackrory, can you now help us with the debian side of things? we know nothing about debian packaging over here.

          Show
          jayunit100 jay vyas added a comment - And in isolation, tachyon is definetly working. [root@bigtop1 ~]# tachyon runTest Basic MUST_CACHE /BasicFile_MUST_CACHE has been removed 2014-11-14 23:48:33,188 INFO (TachyonFS.java:connect) - Trying to connect master @ localhost/127.0.0.1:19998 2014-11-14 23:48:33,242 INFO (MasterClient.java:getUserId) - User registered at the master localhost/127.0.0.1:19998 got UserId 7 2014-11-14 23:48:33,243 INFO (TachyonFS.java:connect) - Trying to get local worker host : bigtop1.vagrant 2014-11-14 23:48:33,254 INFO (TachyonFS.java:connect) - Connecting local worker @ bigtop1.vagrant/10.10.10.11:29998 2014-11-14 23:48:33,282 INFO (CommonUtils.java:printTimeTakenMs) - createFile with fileId 4 took 96 ms. 2014-11-14 23:48:33,320 INFO (TachyonFS.java:createAndGetUserTempFolder) - Folder /mnt/ramdisk/tachyonworker/users/7 was created! 2014-11-14 23:48:33,331 INFO (BlockOutStream.java:<init>) - /mnt/ramdisk/tachyonworker/users/7/4294967296 was created! Passed the test! Sean Mackrory , can you now help us with the debian side of things? we know nothing about debian packaging over here.
          Hide
          cos Konstantin Boudnik added a comment -

          we know nothing about debian packaging over here

          I am shocked! SHOCKED!

          Show
          cos Konstantin Boudnik added a comment - we know nothing about debian packaging over here I am shocked! SHOCKED!
          Hide
          dcapwell David Capwell added a comment -

          Awesome to see this!

          Also, you don't know deb or Mac? You keep failing me son!

          Show
          dcapwell David Capwell added a comment - Awesome to see this! Also, you don't know deb or Mac? You keep failing me son!
          Hide
          plinnell Peter Linnell added a comment -

          Can we get a patch posted here to review and test ?

          Show
          plinnell Peter Linnell added a comment - Can we get a patch posted here to review and test ?
          Hide
          hsaputra Henry Saputra added a comment -

          From Cos' comment in BIGTOP-1493, should the patch for BIGTOP-1493 go in first then?

          Show
          hsaputra Henry Saputra added a comment - From Cos' comment in BIGTOP-1493 , should the patch for BIGTOP-1493 go in first then?
          Hide
          jayunit100 jay vyas added a comment -
          Show
          jayunit100 jay vyas added a comment - Henry Saputra Saptura I'm okay with committing it bef, but i believe cos and roman both wanted it in after . Peter Linnell Yup Huamin Chen can make the patch and submit it tomorrow i guess, as mentioned above, we don't have deb packaging yet though, so we will need help w/ that. the code is here : https://github.com/rootfs/bigtop/compare/tachyon?expand=1 if you want to check it out .
          Hide
          chenh Huamin Chen added a comment -

          Add Tachyon package and rpm build spec.

          Show
          chenh Huamin Chen added a comment - Add Tachyon package and rpm build spec.
          Hide
          chenh Huamin Chen added a comment -

          Is deb pkg required for this patch?

          Show
          chenh Huamin Chen added a comment - Is deb pkg required for this patch?
          Hide
          cos Konstantin Boudnik added a comment -

          Yes, please.

          Show
          cos Konstantin Boudnik added a comment - Yes, please.
          Hide
          chenh Huamin Chen added a comment -

          OK, will update the patch with deb pkg.

          Show
          chenh Huamin Chen added a comment - OK, will update the patch with deb pkg.
          Hide
          jayunit100 jay vyas added a comment -

          Yup Huamin Chen, deb is required ... FYI now that our first packaging adventure is coming to closure, ive updated the wiki page https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+Packaging . Once you get deb working let me know how (im also not familiar w/ it) and I'll update the packaging page so there is a blueprint to follow for others.

          Show
          jayunit100 jay vyas added a comment - Yup Huamin Chen , deb is required ... FYI now that our first packaging adventure is coming to closure, ive updated the wiki page https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+Packaging . Once you get deb working let me know how (im also not familiar w/ it) and I'll update the packaging page so there is a blueprint to follow for others.
          Hide
          chenh Huamin Chen added a comment -

          Add second patch set that includes deb packaging.

          Show
          chenh Huamin Chen added a comment - Add second patch set that includes deb packaging.
          Hide
          chenh Huamin Chen added a comment -

          Add tachyon to bigtop

          Show
          chenh Huamin Chen added a comment - Add tachyon to bigtop
          Hide
          jayunit100 jay vyas added a comment -
          • Ah One last thing, iirc for bigtop we require to add a puppet recipe for new components,... i can look into that now.
          • otherwise, it works for me: ran gradle-rpm and it definetly build the right stuff. Some minor whitespace issues that i can squelch on commit. I'll give others some time to review also.
          /etc/tachyon/slaves
          /etc/tachyon/tachyon-env.sh
          /usr/bin/tachyon
          /usr/lib/tachyon/bin/tachyon
          /usr/lib/tachyon/bin/tachyon-mount.sh
          /usr/lib/tachyon/bin/tachyon-slaves.sh
          /usr/lib/tachyon/bin/tachyon-start.sh
          /usr/lib/tachyon/bin/tachyon-stop.sh
          /usr/lib/tachyon/libexec/tachyon-config.sh
          /usr/lib/tachyon/libexec/tachyon-layout.sh
          /usr/lib/tachyon/libexec/tachyon-layout.sh.linux.template
          /usr/lib/tachyon/tachyon-0.5.0-jar-with-dependencies.jar
          /usr/lib/tachyon/tachyon-0.5.0-tests.jar
          /usr/lib/tachyon/tachyon-0.5.0.jar
          /usr/share/doc/tachyon
          /usr/share/doc/tachyon/LICENSE
          /usr/share/doc/tachyon/README.md
          /usr/share/tachyon
          /usr/share/tachyon/web
          /usr/share/tachyon/web/webapp
          /usr/share/tachyon/web/webapp/browse.jsp
          /usr/share/tachyon/web/webapp/configuration.jsp
          /usr/share/tachyon/web/webapp/css
          /usr/share/tachyon/web/webapp/css/bootstrap-responsive.min.css
          /usr/share/tachyon/web/webapp/css/bootstrap.min.css
          /usr/share/tachyon/web/webapp/css/tachyoncustom.min.css
          /usr/share/tachyon/web/webapp/dependency.jsp
          /usr/share/tachyon/web/webapp/general.jsp
          /usr/share/tachyon/web/webapp/img
          /usr/share/tachyon/web/webapp/img/amplab_logo.png
          /usr/share/tachyon/web/webapp/img/glyphicons-halflings-white.png
          /usr/share/tachyon/web/webapp/img/glyphicons-halflings.png
          /usr/share/tachyon/web/webapp/img/logo.png
          /usr/share/tachyon/web/webapp/index.html
          /usr/share/tachyon/web/webapp/js
          /usr/share/tachyon/web/webapp/js/bootstrap.min.js
          /usr/share/tachyon/web/webapp/js/jquery-1.9.1.min.js
          /usr/share/tachyon/web/webapp/memory.jsp
          /usr/share/tachyon/web/webapp/viewFile.jsp
          
          Show
          jayunit100 jay vyas added a comment - Ah One last thing, iirc for bigtop we require to add a puppet recipe for new components,... i can look into that now. otherwise, it works for me: ran gradle-rpm and it definetly build the right stuff. Some minor whitespace issues that i can squelch on commit. I'll give others some time to review also. /etc/tachyon/slaves /etc/tachyon/tachyon-env.sh /usr/bin/tachyon /usr/lib/tachyon/bin/tachyon /usr/lib/tachyon/bin/tachyon-mount.sh /usr/lib/tachyon/bin/tachyon-slaves.sh /usr/lib/tachyon/bin/tachyon-start.sh /usr/lib/tachyon/bin/tachyon-stop.sh /usr/lib/tachyon/libexec/tachyon-config.sh /usr/lib/tachyon/libexec/tachyon-layout.sh /usr/lib/tachyon/libexec/tachyon-layout.sh.linux.template /usr/lib/tachyon/tachyon-0.5.0-jar-with-dependencies.jar /usr/lib/tachyon/tachyon-0.5.0-tests.jar /usr/lib/tachyon/tachyon-0.5.0.jar /usr/share/doc/tachyon /usr/share/doc/tachyon/LICENSE /usr/share/doc/tachyon/README.md /usr/share/tachyon /usr/share/tachyon/web /usr/share/tachyon/web/webapp /usr/share/tachyon/web/webapp/browse.jsp /usr/share/tachyon/web/webapp/configuration.jsp /usr/share/tachyon/web/webapp/css /usr/share/tachyon/web/webapp/css/bootstrap-responsive.min.css /usr/share/tachyon/web/webapp/css/bootstrap.min.css /usr/share/tachyon/web/webapp/css/tachyoncustom.min.css /usr/share/tachyon/web/webapp/dependency.jsp /usr/share/tachyon/web/webapp/general.jsp /usr/share/tachyon/web/webapp/img /usr/share/tachyon/web/webapp/img/amplab_logo.png /usr/share/tachyon/web/webapp/img/glyphicons-halflings-white.png /usr/share/tachyon/web/webapp/img/glyphicons-halflings.png /usr/share/tachyon/web/webapp/img/logo.png /usr/share/tachyon/web/webapp/index.html /usr/share/tachyon/web/webapp/js /usr/share/tachyon/web/webapp/js/bootstrap.min.js /usr/share/tachyon/web/webapp/js/jquery-1.9.1.min.js /usr/share/tachyon/web/webapp/memory.jsp /usr/share/tachyon/web/webapp/viewFile.jsp
          Hide
          dcapwell David Capwell added a comment -

          Hmmm where is the client uber jar? Mr and spark should use that jar and not
          the one in the list above (server uber jar)

          Show
          dcapwell David Capwell added a comment - Hmmm where is the client uber jar? Mr and spark should use that jar and not the one in the list above (server uber jar)
          Hide
          jayunit100 jay vyas added a comment - - edited

          FYI, testing puppet stuff now. I added it to huamin's branch on github. There is a nice vagrant workflow for this.

          • ON YOUR HOST:
            vagrant up to create a vm from vm/bigtop-deploy/vagrant-puppet, then
            gradlew tacyhon-yum to create a yum repo for the tachyon package from source
            then vagrant ssh
          • Now, ON THE VM:
            run sudo yum-config-manager --add-repo file:///bigtop-home/output/ .

          The result is that now you can test puppet recipes against your locally created bigtop packages, by just running puppet apply -d --confdir=/bigtop-home/bigtop-deploy/puppet/ --modulepath=/bigtop-home/bigtop-deploy/puppet/modules /bigtop-home/bigtop-deploy/puppet/manifests/site.pp inside of your VM.

          So, you can iteratively rebuild the package and update puppet recipes in your host, and just run the puupet command in the VM to develop bigtop packaging and puppet code. It actually makes puppet integration fun Not anything novel, but just making a note of this for those hacking on this. will leave update shortly.

          Show
          jayunit100 jay vyas added a comment - - edited FYI, testing puppet stuff now. I added it to huamin's branch on github. There is a nice vagrant workflow for this. ON YOUR HOST: vagrant up to create a vm from vm/bigtop-deploy/vagrant-puppet, then gradlew tacyhon-yum to create a yum repo for the tachyon package from source then vagrant ssh Now, ON THE VM: run sudo yum-config-manager --add-repo file:///bigtop-home/output/ . The result is that now you can test puppet recipes against your locally created bigtop packages, by just running puppet apply -d --confdir=/bigtop-home/bigtop-deploy/puppet/ --modulepath=/bigtop-home/bigtop-deploy/puppet/modules /bigtop-home/bigtop-deploy/puppet/manifests/site.pp inside of your VM. So, you can iteratively rebuild the package and update puppet recipes in your host, and just run the puupet command in the VM to develop bigtop packaging and puppet code. It actually makes puppet integration fun Not anything novel, but just making a note of this for those hacking on this. will leave update shortly.
          Hide
          jayunit100 jay vyas added a comment - - edited

          Huamin Chen David Capwell okay guys ! updated huamin's tachyon branch w/ working puppet recipes for tachyon. feel free to clean up, rebase, and then huamin just go ahead and submit a patch and I'll review it from scratch ! https://github.com/rootfs/bigtop/commits/tachyon . we are in the home stretch ! We can do a follow on JIRA to fine tune the puppet recipes some more.

          Show
          jayunit100 jay vyas added a comment - - edited Huamin Chen David Capwell okay guys ! updated huamin's tachyon branch w/ working puppet recipes for tachyon. feel free to clean up, rebase, and then huamin just go ahead and submit a patch and I'll review it from scratch ! https://github.com/rootfs/bigtop/commits/tachyon . we are in the home stretch ! We can do a follow on JIRA to fine tune the puppet recipes some more.
          Hide
          chenh Huamin Chen added a comment -

          Patches include

          • RPM build spec
          • Deb build
          • init_d scripts
          • puppet recipe
          Show
          chenh Huamin Chen added a comment - Patches include RPM build spec Deb build init_d scripts puppet recipe
          Hide
          chenh Huamin Chen added a comment -

          Thanks jay vyas, patch uploaded.

          Show
          chenh Huamin Chen added a comment - Thanks jay vyas , patch uploaded.
          Hide
          jayunit100 jay vyas added a comment -

          thanks Huamin Chen . testing now...

          Show
          jayunit100 jay vyas added a comment - thanks Huamin Chen . testing now...
          Hide
          jayunit100 jay vyas added a comment -

          reattached w/ squash commits

          Show
          jayunit100 jay vyas added a comment - reattached w/ squash commits
          Hide
          chenh Huamin Chen added a comment -

          new patch, squashed and rebased

          Show
          chenh Huamin Chen added a comment - new patch, squashed and rebased
          Hide
          jayunit100 jay vyas added a comment - - edited

          +1 tested and commited..... Tachyon is now a part of apache bigtop !
          thanks Huamin Chen, David Capwell , Henry Saputra , Roman Shaposhnik Konstantin Boudnik and Sean Mackrory for all your help on this. !

          Now on to adding smoke tests and better vagrant integration so users can test tachyon with the click of a button !

          Show
          jayunit100 jay vyas added a comment - - edited +1 tested and commited..... Tachyon is now a part of apache bigtop ! thanks Huamin Chen , David Capwell , Henry Saputra , Roman Shaposhnik Konstantin Boudnik and Sean Mackrory for all your help on this. ! Now on to adding smoke tests and better vagrant integration so users can test tachyon with the click of a button !
          Hide
          jayunit100 jay vyas added a comment -

          (resolving and assigned to myself since hchen isnt coming up in the assignees for some reason)

          Show
          jayunit100 jay vyas added a comment - (resolving and assigned to myself since hchen isnt coming up in the assignees for some reason)
          Hide
          chenh Huamin Chen added a comment -

          jay vyas thanks!

          Show
          chenh Huamin Chen added a comment - jay vyas thanks!
          Hide
          dcapwell David Capwell added a comment -

          Show
          dcapwell David Capwell added a comment -
          Hide
          hsaputra Henry Saputra added a comment -

          YOU ARE THE MAN!

          Thanks jay vyas and Huamin Chen for all the hard work.

          Thanks guys! Remind me to buy you beer when you guys come to visit bay area.

          Show
          hsaputra Henry Saputra added a comment - YOU ARE THE MAN! Thanks jay vyas and Huamin Chen for all the hard work. Thanks guys! Remind me to buy you beer when you guys come to visit bay area.
          Hide
          cos Konstantin Boudnik added a comment -

          How come this one is marked against 1.0? We haven't even started working on it... Changing to 0.9

          Show
          cos Konstantin Boudnik added a comment - How come this one is marked against 1.0? We haven't even started working on it... Changing to 0.9
          Hide
          jayunit100 jay vyas added a comment -

          Hi cos... Tachyon packaging has definitely completed the first iteration and is in bigtop master, What do u mean?

          0.9 is ok, but fyi current tachyon 5.0 is buggy
          And it won't be easy to get interop till 6.0 is released iirc

          Show
          jayunit100 jay vyas added a comment - Hi cos... Tachyon packaging has definitely completed the first iteration and is in bigtop master, What do u mean? 0.9 is ok, but fyi current tachyon 5.0 is buggy And it won't be easy to get interop till 6.0 is released iirc
          Hide
          cos Konstantin Boudnik added a comment -

          I meant release of Bigtop that ticket has been marked as fixed in.

          Show
          cos Konstantin Boudnik added a comment - I meant release of Bigtop that ticket has been marked as fixed in.

            People

            • Assignee:
              jayunit100 jay vyas
              Reporter:
              jayunit100 jay vyas
            • Votes:
              1 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development