Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.1.0
    • Component/s: general
    • Labels:
      None

      Description

      Zeppelin will moves its repository into ASF. I think Zeppelin needs to become a part of Bigtop.

      1. add-zepp-tobom.patch
        1 kB
        Konstantin Boudnik
      2. BIGTOP-1769.patch
        38 kB
        Konstantin Boudnik
      3. BIGTOP-1769.patch
        28 kB
        Jonathan Kelly
      4. BIGTOP-1769.patch
        30 kB
        Jonathan Kelly
      5. cluster.yaml.rej
        1 kB
        Konstantin Boudnik

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user ejono closed the pull request at:

          https://github.com/apache/bigtop/pull/44

          Show
          githubbot ASF GitHub Bot added a comment - Github user ejono closed the pull request at: https://github.com/apache/bigtop/pull/44
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user c0s commented on the issue:

          https://github.com/apache/bigtop/pull/44

          I guess...

          Show
          githubbot ASF GitHub Bot added a comment - Github user c0s commented on the issue: https://github.com/apache/bigtop/pull/44 I guess...
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user bzz commented on the issue:

          https://github.com/apache/bigtop/pull/44

          Shall this one be closed, as soon as BIGTOP-1769(https://issues.apache.org/jira/browse/BIGTOP-1769) is resolved?

          Show
          githubbot ASF GitHub Bot added a comment - Github user bzz commented on the issue: https://github.com/apache/bigtop/pull/44 Shall this one be closed, as soon as BIGTOP-1769 ( https://issues.apache.org/jira/browse/BIGTOP-1769 ) is resolved?
          Hide
          cos Konstantin Boudnik added a comment -

          Resolving with a couple of additional issue which would have to affected in 1.2. Thanks!

          Show
          cos Konstantin Boudnik added a comment - Resolving with a couple of additional issue which would have to affected in 1.2. Thanks!
          Hide
          cos Konstantin Boudnik added a comment -

          Sorry, I am dense today - they are cutting trees outside of my house and it drives me nuts - but could you elaborate a little bit on this point?

          Show
          cos Konstantin Boudnik added a comment - Sorry, I am dense today - they are cutting trees outside of my house and it drives me nuts - but could you elaborate a little bit on this point?
          Hide
          rvs Roman Shaposhnik added a comment -

          Konstantin Boudnik sorry for coming to the party late, but why not junction packages zeppelin-foo to take care of plugins AND dependencies at the same time?

          Show
          rvs Roman Shaposhnik added a comment - Konstantin Boudnik sorry for coming to the party late, but why not junction packages zeppelin-foo to take care of plugins AND dependencies at the same time?
          Hide
          cos Konstantin Boudnik added a comment -

          I am also going to split this into smaller patches, similar to BIGTOP-1944. Otherwise, the commits would be huge and somewhat unrelated.

          Show
          cos Konstantin Boudnik added a comment - I am also going to split this into smaller patches, similar to BIGTOP-1944 . Otherwise, the commits would be huge and somewhat unrelated.
          Hide
          kaiyzen Nate DAmico added a comment -

          If end goal to would be to do the path that was taken with ignite, seems making zeppelin depend on zeppelin interpreters would be a good incremental step to start with

          Show
          kaiyzen Nate DAmico added a comment - If end goal to would be to do the path that was taken with ignite, seems making zeppelin depend on zeppelin interpreters would be a good incremental step to start with
          Hide
          cos Konstantin Boudnik added a comment -

          I still have no good answer on how to solve the dependency inclusion problem here. Some thoughts:

          • making zeppelin package to depend on the packages for zeppelin's interpreters, e.g. hive, kylin, ignite, spark, etc. Once these packages are installed, we can safely add the needed locations into zeppelin classpath, or create a bunch of the softlinks under /usr/lib/zeppelin/interpreter/<name>. This approach allows to guarantee that Zeppelin will be functional even if its node didn't have interpreter components in place. The downside: such node will have all the services installed as well, and potentially can mess-up something.
            • this problem can be solved down the road by changing our packages to universally provide <name-libs>> which will only contain the component's jars in it; and deliver services separately (similar to what has been done in ingite component}}
          • keep the huge package for now (after all we already have ridiculously sized spark-master). In this case we might potentially face the situation where an interpreter libs within Zeppelin aren't the same as the interpreter's component package deliver.

          What are the people here think about this?

          Show
          cos Konstantin Boudnik added a comment - I still have no good answer on how to solve the dependency inclusion problem here. Some thoughts: making zeppelin package to depend on the packages for zeppelin's interpreters, e.g. hive, kylin, ignite, spark, etc. Once these packages are installed, we can safely add the needed locations into zeppelin classpath, or create a bunch of the softlinks under /usr/lib/zeppelin/interpreter/<name> . This approach allows to guarantee that Zeppelin will be functional even if its node didn't have interpreter components in place. The downside: such node will have all the services installed as well, and potentially can mess-up something. this problem can be solved down the road by changing our packages to universally provide <name-libs>> which will only contain the component's jars in it; and deliver services separately (similar to what has been done in ingite component}} keep the huge package for now (after all we already have ridiculously sized spark-master). In this case we might potentially face the situation where an interpreter libs within Zeppelin aren't the same as the interpreter's component package deliver. What are the people here think about this?
          Hide
          cos Konstantin Boudnik added a comment - - edited

          I haven't heard from the author of this patch for a while now. Jonathan Kelly, are you still working on these?

          Show
          cos Konstantin Boudnik added a comment - - edited I haven't heard from the author of this patch for a while now. Jonathan Kelly , are you still working on these?
          Hide
          cos Konstantin Boudnik added a comment - - edited

          Ok, here's the patch that builds both on deb and rpm architecture. I have resolved the conflict in the cluster.yaml to the best of my knowledge but I won't be really surprised if I messed something up there.

          Now, the main problem with the current packaging approach is that it does redistribute ALL dependencies for every single component zeppelin is integrated with. As the result the package comes to about 0.5GB. This is unacceptable for two reasons:

          • the size (lesser of my concerns)
          • an uncontrollable explosion of who-knows-what versions of the libraries pulled in during the build. I have already opened up ZEPPELIN-404 because of the bunch of stuff being pulled in from Cloudera's repos. And similar to the build of Kite (as outlined in here http://is.gd/vxQtgf) we should be signing off releases with the content unknown to us.

          As the fix for the latter we should be setting up softlinks to the components provided by Bigtop stack. Thoughts?

          Show
          cos Konstantin Boudnik added a comment - - edited Ok, here's the patch that builds both on deb and rpm architecture. I have resolved the conflict in the cluster.yaml to the best of my knowledge but I won't be really surprised if I messed something up there. Now, the main problem with the current packaging approach is that it does redistribute ALL dependencies for every single component zeppelin is integrated with. As the result the package comes to about 0.5GB. This is unacceptable for two reasons: the size (lesser of my concerns) an uncontrollable explosion of who-knows-what versions of the libraries pulled in during the build. I have already opened up ZEPPELIN-404 because of the bunch of stuff being pulled in from Cloudera's repos. And similar to the build of Kite (as outlined in here http://is.gd/vxQtgf ) we should be signing off releases with the content unknown to us. As the fix for the latter we should be setting up softlinks to the components provided by Bigtop stack. Thoughts?
          Hide
          cos Konstantin Boudnik added a comment -

          The profile warning is gone if I move the build to upcoming Zeppelin 0.5.5. The package issue is still around though

          Show
          cos Konstantin Boudnik added a comment - The profile warning is gone if I move the build to upcoming Zeppelin 0.5.5. The package issue is still around though
          Hide
          cos Konstantin Boudnik added a comment -

          Patch isn't ready

          Show
          cos Konstantin Boudnik added a comment - Patch isn't ready
          Hide
          cos Konstantin Boudnik added a comment -

          Also, I would recommend to look at the work Olaf Flebbe has done in BIGTOP-2097 and change the deb package in the same way as he did for Ignite. There's no point of dragging forward the unholy packaging stuff we had in place originally.

          Show
          cos Konstantin Boudnik added a comment - Also, I would recommend to look at the work Olaf Flebbe has done in BIGTOP-2097 and change the deb package in the same way as he did for Ignite. There's no point of dragging forward the unholy packaging stuff we had in place originally.
          Hide
          cos Konstantin Boudnik added a comment -

          Now, I was running this build against the current master, which has already moved to Spark 1.5.1. Here are a few issues:

          • might be fixed in coming 0.5.5 release? I would love to try it out, please give me any pointers
            [WARNING] The requested profile "spark-1.5" could not be activated because it does not exist.
            make[1]: Leaving directory `/ws/output/zeppelin/zeppelin-0.5.0'
            
          • package build fails like this
            cp: cannot stat 'debian/tmp//usr/lib/zeppelin/conf/': No such file or directory
            dh_install: cp -a debian/tmp//usr/lib/zeppelin/conf/ debian/zeppelin///usr/lib/zeppelin/ returned exit code 1
            make: *** [binary] Error 2
            dpkg-buildpackage: error: fakeroot debian/rules binary gave error exit status 2
            
          Show
          cos Konstantin Boudnik added a comment - Now, I was running this build against the current master, which has already moved to Spark 1.5.1. Here are a few issues: might be fixed in coming 0.5.5 release? I would love to try it out, please give me any pointers [WARNING] The requested profile "spark-1.5" could not be activated because it does not exist. make[1]: Leaving directory `/ws/output/zeppelin/zeppelin-0.5.0' package build fails like this cp: cannot stat 'debian/tmp//usr/lib/zeppelin/conf/': No such file or directory dh_install: cp -a debian/tmp//usr/lib/zeppelin/conf/ debian/zeppelin///usr/lib/zeppelin/ returned exit code 1 make: *** [binary] Error 2 dpkg-buildpackage: error: fakeroot debian/rules binary gave error exit status 2
          Hide
          cos Konstantin Boudnik added a comment - - edited

          I am seeing a lot of ERRORs from npm during the build, which I am running in as a non-privileged user. Shall the build be executed as root? Cause that might be an issue... The zeppelin build succeeded though.

          Show
          cos Konstantin Boudnik added a comment - - edited I am seeing a lot of ERRORs from npm during the build, which I am running in as a non-privileged user. Shall the build be executed as root? Cause that might be an issue... The zeppelin build succeeded though.
          Hide
          cos Konstantin Boudnik added a comment -

          Here's the patch for adding zeppelin to the bigtop.bom

          Show
          cos Konstantin Boudnik added a comment - Here's the patch for adding zeppelin to the bigtop.bom
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user c0s commented on the pull request:

          https://github.com/apache/bigtop/pull/44#issuecomment-154512299

          Another issue with this patch: how I suppose to build it? The definition of the new component needs to be added to the bigtop.bom file

          Show
          githubbot ASF GitHub Bot added a comment - Github user c0s commented on the pull request: https://github.com/apache/bigtop/pull/44#issuecomment-154512299 Another issue with this patch: how I suppose to build it? The definition of the new component needs to be added to the bigtop.bom file
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user c0s commented on the pull request:

          https://github.com/apache/bigtop/pull/44#issuecomment-154510843

          Patch doesn't apply to the current master anymore. Could you please fix/rebase it again? Thanks!

          Show
          githubbot ASF GitHub Bot added a comment - Github user c0s commented on the pull request: https://github.com/apache/bigtop/pull/44#issuecomment-154510843 Patch doesn't apply to the current master anymore. Could you please fix/rebase it again? Thanks!
          Hide
          cos Konstantin Boudnik added a comment -

          The patch doesn't apply cleanly anymore. Jonathan Kelly, could you please take a look? Thanks!

          Show
          cos Konstantin Boudnik added a comment - The patch doesn't apply cleanly anymore. Jonathan Kelly , could you please take a look? Thanks!
          Hide
          cos Konstantin Boudnik added a comment -

          Ok now we have BIGTOP-2104 in place, which adds Spark 1.5.1. Jonathan Kelly, do you think we can proceed with the Zeppelin integration?

          Show
          cos Konstantin Boudnik added a comment - Ok now we have BIGTOP-2104 in place, which adds Spark 1.5.1. Jonathan Kelly , do you think we can proceed with the Zeppelin integration?
          Hide
          cos Konstantin Boudnik added a comment -

          Thanks for the info, Jonathan Kelly. I have commented elsewhere, but to me blocking Spark upgrade on Hive server. If anything, Spark is an execution engine, and Hive is merely a query planner.

          Show
          cos Konstantin Boudnik added a comment - Thanks for the info, Jonathan Kelly . I have commented elsewhere, but to me blocking Spark upgrade on Hive server. If anything, Spark is an execution engine, and Hive is merely a query planner.
          Hide
          jonathak Jonathan Kelly added a comment -

          I created a pull request a couple weeks ago but haven't received any feedback yet. Then again, it is somewhat blocked by (or at least depends on) BIGTOP-1944 (Upgrade Spark), which is blocked by BIGTOP-1792 (Add hive-server2 and hive-metastore services to bigtop hive deployment), which is blocked by us needing to contribute back another patch for contributing back bigtop-mysql (no JIRA for this yet, AFAIK).

          Show
          jonathak Jonathan Kelly added a comment - I created a pull request a couple weeks ago but haven't received any feedback yet. Then again, it is somewhat blocked by (or at least depends on) BIGTOP-1944 (Upgrade Spark), which is blocked by BIGTOP-1792 (Add hive-server2 and hive-metastore services to bigtop hive deployment), which is blocked by us needing to contribute back another patch for contributing back bigtop-mysql (no JIRA for this yet, AFAIK).
          Hide
          cos Konstantin Boudnik added a comment -

          I wonder where are we with this? Anything I can do to help it move to the closure?

          Show
          cos Konstantin Boudnik added a comment - I wonder where are we with this? Anything I can do to help it move to the closure?
          Hide
          jonathak Jonathan Kelly added a comment -

          YoungWoo Kim, sorry, but I have not yet had a chance to look at your patch, but as you can see below, I've created a pull request for Zeppelin integration. The pull request does not currently incorporate any of the changes you made to my patch, but I'll try to find time to do so later, unless you have time. Thanks you again!

          Show
          jonathak Jonathan Kelly added a comment - YoungWoo Kim , sorry, but I have not yet had a chance to look at your patch, but as you can see below, I've created a pull request for Zeppelin integration. The pull request does not currently incorporate any of the changes you made to my patch, but I'll try to find time to do so later, unless you have time. Thanks you again!
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user ejono opened a pull request:

          https://github.com/apache/bigtop/pull/44

          BIGTOP-1769. Zeppelin Integration

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/ejono/bigtop BIGTOP-1769

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/bigtop/pull/44.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #44


          commit 1e7c2c111bd30a05b2b0e58a8c53e5cb268b9506
          Author: Jonathan Kelly <jonathak@amazon.com>
          Date: 2015-09-29T23:01:26Z

          BIGTOP-1769. Zeppelin Integration


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user ejono opened a pull request: https://github.com/apache/bigtop/pull/44 BIGTOP-1769 . Zeppelin Integration You can merge this pull request into a Git repository by running: $ git pull https://github.com/ejono/bigtop BIGTOP-1769 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/44.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #44 commit 1e7c2c111bd30a05b2b0e58a8c53e5cb268b9506 Author: Jonathan Kelly <jonathak@amazon.com> Date: 2015-09-29T23:01:26Z BIGTOP-1769 . Zeppelin Integration
          Hide
          bzz Alexander Bezzubov added a comment -

          Thank you, makes perfect sense! It would be awesome if you could look into this issue.

          Show
          bzz Alexander Bezzubov added a comment - Thank you, makes perfect sense! It would be awesome if you could look into this issue.
          Hide
          jongyoul Jongyoul Lee added a comment -

          Alexander Bezzubov, Jonathan Kelly has an ownership for this issue after he submit the patch at first. I think his opinion is important. If he doesn't have enough time to contribute this, I'm willing to handle this issue.

          Show
          jongyoul Jongyoul Lee added a comment - Alexander Bezzubov , Jonathan Kelly has an ownership for this issue after he submit the patch at first. I think his opinion is important. If he doesn't have enough time to contribute this, I'm willing to handle this issue.
          Hide
          bzz Alexander Bezzubov added a comment - - edited

          jay vyas right,
          YoungWoo Kim Jonathan Kelly Jongyoul Lee great work have been done so far integrating Zeppelin to BigTop, any chances you guys have some spare cycles to look into it again to close this down?
          Please, also let me know if you need any help here.

          Show
          bzz Alexander Bezzubov added a comment - - edited jay vyas right, YoungWoo Kim Jonathan Kelly Jongyoul Lee great work have been done so far integrating Zeppelin to BigTop, any chances you guys have some spare cycles to look into it again to close this down? Please, also let me know if you need any help here.
          Hide
          jayunit100 jay vyas added a comment -

          Hi alexander , just saw this. I think someone needs to take ownership and polish it off, looks like jonathan/ youngwoo was pretty close ?

          Show
          jayunit100 jay vyas added a comment - Hi alexander , just saw this. I think someone needs to take ownership and polish it off, looks like jonathan/ youngwoo was pretty close ?
          Hide
          bzz Alexander Bezzubov added a comment -

          Guys, is there anything that can be done from the Zepplin side to help with this patch landing?

          Show
          bzz Alexander Bezzubov added a comment - Guys, is there anything that can be done from the Zepplin side to help with this patch landing?
          Hide
          warwithin YoungWoo Kim added a comment -

          Thanks for the update, Jonathan Kelly! It would be nice to have your complete patch for users. So, I'll make an effort for getting them merged.

          Show
          warwithin YoungWoo Kim added a comment - Thanks for the update, Jonathan Kelly ! It would be nice to have your complete patch for users. So, I'll make an effort for getting them merged.
          Hide
          jonathak Jonathan Kelly added a comment -

          Thanks a lot, YoungWoo Kim, that's awesome. I haven't gotten a chance to take a look at it yet though because I've been busy with other things.

          Also, I realized that it's difficult for me to provide my latest patch for this (which includes support for Puppet deployment) until we can get our large patches for BIGTOP-1746 and BIGTOP-1689 merged in as well.

          Show
          jonathak Jonathan Kelly added a comment - Thanks a lot, YoungWoo Kim , that's awesome. I haven't gotten a chance to take a look at it yet though because I've been busy with other things. Also, I realized that it's difficult for me to provide my latest patch for this (which includes support for Puppet deployment) until we can get our large patches for BIGTOP-1746 and BIGTOP-1689 merged in as well.
          Hide
          warwithin YoungWoo Kim added a comment - - edited

          Jonathan Kelly, If you don't mind, I would like to send a PR which is a revised patch from internal branch based on your patch. I did not touch the authorship information and what I did:

          • resolved merge conflicts
          • updated BUILD_OPTS for Hive
          • added symlinks for Hadoop, Hive and Zookeeper jars
          • updated versions related deb (minor)

          Commit is here, https://github.com/youngwookim/bigtop/commit/5fb3f2ec4d923029965934e6e91a98d3351ca7f5

          And packaging works on both, rpm and deb. I tested it manually through the Z interpreters: spark, hive, md, sh.

          Show
          warwithin YoungWoo Kim added a comment - - edited Jonathan Kelly , If you don't mind, I would like to send a PR which is a revised patch from internal branch based on your patch. I did not touch the authorship information and what I did: resolved merge conflicts updated BUILD_OPTS for Hive added symlinks for Hadoop, Hive and Zookeeper jars updated versions related deb (minor) Commit is here, https://github.com/youngwookim/bigtop/commit/5fb3f2ec4d923029965934e6e91a98d3351ca7f5 And packaging works on both, rpm and deb. I tested it manually through the Z interpreters: spark, hive, md, sh.
          Hide
          warwithin YoungWoo Kim added a comment -

          Apache Zeppelin team recently released 0.5.0-incubating!, https://www.mail-archive.com/general@incubator.apache.org/msg49707.html

          Show
          warwithin YoungWoo Kim added a comment - Apache Zeppelin team recently released 0.5.0-incubating!, https://www.mail-archive.com/general@incubator.apache.org/msg49707.html
          Hide
          kaiyzen Nate DAmico added a comment -

          Seems few weeks out post 1.0 and after zeppelin pushes is 0.5x release will be good timing

          Show
          kaiyzen Nate DAmico added a comment - Seems few weeks out post 1.0 and after zeppelin pushes is 0.5x release will be good timing
          Hide
          jonathak Jonathan Kelly added a comment -

          Sorry this has been sitting for a while. I hope to pick this back up within the next few weeks.

          Show
          jonathak Jonathan Kelly added a comment - Sorry this has been sitting for a while. I hope to pick this back up within the next few weeks.
          Hide
          warwithin YoungWoo Kim added a comment -

          Makes sense. Will look into new patch. Thanks!

          Show
          warwithin YoungWoo Kim added a comment - Makes sense. Will look into new patch. Thanks!
          Hide
          jonathak Jonathan Kelly added a comment -

          OK, I have updated the patch. One more thing that needs to be done before this is committed is to address BIGTOP-1795.

          Show
          jonathak Jonathan Kelly added a comment - OK, I have updated the patch. One more thing that needs to be done before this is committed is to address BIGTOP-1795 .
          Hide
          jonathak Jonathan Kelly added a comment - - edited

          Actually, I decided to remove interpreter.json from the RPM, though I'm still generating it in the Puppet deploy scripts (not part of this JIRA). Once I remove this file from this patch, the patch will no longer reference anything in the "com.nflabs" namespace.

          And as for the Hive interpreter, that's not showing up due to ZEPPELIN-41. I think it would be best to leave ZEPPELIN_INTERPRETERS empty in the zeppelin-env.sh included in the RPM, so that newly added interpreters are enabled by default, without us having to remember to add them here.

          In short, I will update the patch to remove interpreter.json (and also rebase onto latest master).

          Show
          jonathak Jonathan Kelly added a comment - - edited Actually, I decided to remove interpreter.json from the RPM, though I'm still generating it in the Puppet deploy scripts (not part of this JIRA). Once I remove this file from this patch, the patch will no longer reference anything in the "com.nflabs" namespace. And as for the Hive interpreter, that's not showing up due to ZEPPELIN-41 . I think it would be best to leave ZEPPELIN_INTERPRETERS empty in the zeppelin-env.sh included in the RPM, so that newly added interpreters are enabled by default, without us having to remember to add them here. In short, I will update the patch to remove interpreter.json (and also rebase onto latest master).
          Hide
          jonathak Jonathan Kelly added a comment -

          YoungWoo Kim, good catch on the namespace change. I'd meant to handle that a couple days ago when I merged from github/master to our local branch I've been building from, but it completely slipped my mind. I will update them in the patch (in interpreters.json).

          As for the Hive Interpreter, I wasn't aware of it, so thanks for letting me know about it. I tried enabling it though (by adding it to ZEPPELIN_INTERPRETERS in zeppelin-env.sh, which I was not previously setting at all--btw, why isn't this new HiveInterpreter enabled by default? Too new?), but it isn't working for some reason. It still doesn't even show up on the Interpreters page after restarting Zeppelin.

          Show
          jonathak Jonathan Kelly added a comment - YoungWoo Kim , good catch on the namespace change. I'd meant to handle that a couple days ago when I merged from github/master to our local branch I've been building from, but it completely slipped my mind. I will update them in the patch (in interpreters.json). As for the Hive Interpreter, I wasn't aware of it, so thanks for letting me know about it. I tried enabling it though (by adding it to ZEPPELIN_INTERPRETERS in zeppelin-env.sh, which I was not previously setting at all--btw, why isn't this new HiveInterpreter enabled by default? Too new?), but it isn't working for some reason. It still doesn't even show up on the Interpreters page after restarting Zeppelin.
          Hide
          warwithin YoungWoo Kim added a comment -

          Thanks for the update Jonathan! A few things:

          • Looks like namespace for interpreters has been changed on upstream. ZEPPELIN-12
          • interperter.json: A new interpreter for Apache Hive, Link. Should be added?
          Show
          warwithin YoungWoo Kim added a comment - Thanks for the update Jonathan! A few things: Looks like namespace for interpreters has been changed on upstream. ZEPPELIN-12 interperter.json: A new interpreter for Apache Hive, Link . Should be added?
          Hide
          jonathak Jonathan Kelly added a comment -

          I've updated the patch with a fix for the bug with the service not stopping correctly. I'm now using zeppelin-daemon.sh instead of zeppelin.sh to start the process (so the creation of the pif file is now handled by zeppelin-daemon.sh), and I'm setting the PIDFILE in order to match what zeppelin-daemon.sh is using.

          Show
          jonathak Jonathan Kelly added a comment - I've updated the patch with a fix for the bug with the service not stopping correctly. I'm now using zeppelin-daemon.sh instead of zeppelin.sh to start the process (so the creation of the pif file is now handled by zeppelin-daemon.sh), and I'm setting the PIDFILE in order to match what zeppelin-daemon.sh is using.
          Hide
          jonathak Jonathan Kelly added a comment -

          Of course not, that was my concern. Don't worry though, I'm not doing that. It just seemed like some problems that I was having a few weeks ago could have been caused by the fact that I was not doing so, which is what concerned me. That's all been resolved though.

          Show
          jonathak Jonathan Kelly added a comment - Of course not, that was my concern. Don't worry though, I'm not doing that. It just seemed like some problems that I was having a few weeks ago could have been caused by the fact that I was not doing so, which is what concerned me. That's all been resolved though.
          Hide
          plinnell Peter Linnell added a comment -

          JFYI, Building packages as root is never a good idea.

          Show
          plinnell Peter Linnell added a comment - JFYI, Building packages as root is never a good idea.
          Hide
          warwithin YoungWoo Kim added a comment -

          I like the idea using HDFS on tutorial! and let me take a look BIGGTOP-1795. Thanks!

          Show
          warwithin YoungWoo Kim added a comment - I like the idea using HDFS on tutorial! and let me take a look BIGGTOP-1795. Thanks!
          Hide
          jonathak Jonathan Kelly added a comment -

          Thanks for taking a look, YoungWoo Kim!

          Yes, I've recently noticed the same and now have it on my list of things to fix. It has to do with the possibly weird way that the service gets started. The PID in the .pid file is for a bash process that runs zeppelin.sh with nohup, then zeppelin.sh has another child bash process, then that process has the java child process.

          $ cat /var/run/zeppelin/zeppelin.pid
          5101
          $ ps auxwwwf | grep zeppelin
          zeppelin  5101  0.0  0.0   9424   620 ?        S    23:35   0:00 bash -c cd /var/lib/zeppelin && nohup nice -n 0         /usr/lib/zeppelin/bin/zeppelin.sh         > /var/log/zeppelin/zeppelin.out 2>&1 & echo $!
          zeppelin  5102  0.0  0.0   9424  1408 ?        S    23:35   0:00  \_ /bin/bash /usr/lib/zeppelin/bin/zeppelin.sh
          zeppelin  5122  1.3  1.5 2913908 242444 ?      Sl   23:35   0:05      \_ /usr/lib/jvm/java-openjdk/bin/java -cp ...
          
          Show
          jonathak Jonathan Kelly added a comment - Thanks for taking a look, YoungWoo Kim ! Yes, I've recently noticed the same and now have it on my list of things to fix. It has to do with the possibly weird way that the service gets started. The PID in the .pid file is for a bash process that runs zeppelin.sh with nohup, then zeppelin.sh has another child bash process, then that process has the java child process. $ cat /var/run/zeppelin/zeppelin.pid 5101 $ ps auxwwwf | grep zeppelin zeppelin 5101 0.0 0.0 9424 620 ? S 23:35 0:00 bash -c cd /var/lib/zeppelin && nohup nice -n 0 /usr/lib/zeppelin/bin/zeppelin.sh > /var/log/zeppelin/zeppelin.out 2>&1 & echo $! zeppelin 5102 0.0 0.0 9424 1408 ? S 23:35 0:00 \_ /bin/bash /usr/lib/zeppelin/bin/zeppelin.sh zeppelin 5122 1.3 1.5 2913908 242444 ? Sl 23:35 0:05 \_ /usr/lib/jvm/java-openjdk/bin/java -cp ...
          Hide
          warwithin YoungWoo Kim added a comment -

          Jonathan Kelly, I've done a quick check. overall it works fine for me but,

          $ ls -als  /var/run/zeppelin/
          합계 12
          4 drwxr-xr-x   2 zeppelin zeppelin 4096 2015-04-06 11:48 .
          4 drwxr-xr-x. 23 root     root     4096 2015-04-06 11:33 ..
          4 -rw-r--r--   1 zeppelin zeppelin    6 2015-04-06 11:33 zeppelin-interpreter-spark-zeppelin-bigtop1.vagrant.pid
          

          The PID file does not match

          ...
          PIDFILE="/var/run/zeppelin/zeppelin.pid"
          ...
          
          $ sudo service zeppelin stop
          Stopping Zeppelin (zeppelin):                              [  OK  ]
          

          Above command does not work but returns 'OK'.

          Show
          warwithin YoungWoo Kim added a comment - Jonathan Kelly , I've done a quick check. overall it works fine for me but, $ ls -als /var/run/zeppelin/ 합계 12 4 drwxr-xr-x 2 zeppelin zeppelin 4096 2015-04-06 11:48 . 4 drwxr-xr-x. 23 root root 4096 2015-04-06 11:33 .. 4 -rw-r--r-- 1 zeppelin zeppelin 6 2015-04-06 11:33 zeppelin-interpreter-spark-zeppelin-bigtop1.vagrant.pid The PID file does not match ... PIDFILE= "/ var /run/zeppelin/zeppelin.pid" ... $ sudo service zeppelin stop Stopping Zeppelin (zeppelin): [ OK ] Above command does not work but returns 'OK'.
          Hide
          jongyoul Jongyoul Lee added a comment -

          I don't know what was your problem about running spark on yarn exactly, but I think your problem might be setting SPARK_YANR_JAR because your problem had solved after we don't have to set SPARK_YARN_JAR anymore, anyway I think it's good. Second, Zeppelin notebook problem may be solved soon. Finally, as I know, Zeppelin isn't built with Maven 2.x for the first time. Is it?

          Show
          jongyoul Jongyoul Lee added a comment - I don't know what was your problem about running spark on yarn exactly, but I think your problem might be setting SPARK_YANR_JAR because your problem had solved after we don't have to set SPARK_YARN_JAR anymore, anyway I think it's good. Second, Zeppelin notebook problem may be solved soon. Finally, as I know, Zeppelin isn't built with Maven 2.x for the first time. Is it?
          Hide
          jonathak Jonathan Kelly added a comment -

          I now have Zeppelin working with Spark on YARN. I'm not sure exactly what fixed it for me, but there were a few related changes upstream in the Zeppelin project that I pulled in, including no longer requiring SPARK_YARN_JAR, and I also made sure to set SPARK_HOME in zeppelin-env.sh, and now it all works for me.

          There's one minor problem with the Tutorial notebook though, in that it doesn't work well when run on YARN (in fact, it would probably only work correctly in local mode) because it deals with local paths, whereas it would probably be better to use HDFS. But that's something I can bring up in a ZEPPELIN JIRA rather than here.

          One other thing that came up when I pulled in the recent upstream changes in the Zeppelin project is that Maven 3.1+ is now required in order to build Zeppelin, so see the separate JIRA I cut for that: BIGTOP-1795.

          Show
          jonathak Jonathan Kelly added a comment - I now have Zeppelin working with Spark on YARN. I'm not sure exactly what fixed it for me, but there were a few related changes upstream in the Zeppelin project that I pulled in, including no longer requiring SPARK_YARN_JAR, and I also made sure to set SPARK_HOME in zeppelin-env.sh, and now it all works for me. There's one minor problem with the Tutorial notebook though, in that it doesn't work well when run on YARN (in fact, it would probably only work correctly in local mode) because it deals with local paths, whereas it would probably be better to use HDFS. But that's something I can bring up in a ZEPPELIN JIRA rather than here. One other thing that came up when I pulled in the recent upstream changes in the Zeppelin project is that Maven 3.1+ is now required in order to build Zeppelin, so see the separate JIRA I cut for that: BIGTOP-1795 .
          Hide
          cos Konstantin Boudnik added a comment -

          Great! Thanks! Let's do the work on the branch for now so you can be unblocked.
          Please let me know if you need a branch in the Bigtop repo so you can collaborate with others. Thanks!

          Show
          cos Konstantin Boudnik added a comment - Great! Thanks! Let's do the work on the branch for now so you can be unblocked. Please let me know if you need a branch in the Bigtop repo so you can collaborate with others. Thanks!
          Hide
          jonathak Jonathan Kelly added a comment -

          OK, I think it makes sense not to include in the 1.0 release. I have not even figured out yet why I can't get it to work with Spark on YARN. (Haven't gotten any time for that in the past couple of days and might not have any today either.)

          As for maintaining it once it's in Bigtop, I would be happy to do so.

          Show
          jonathak Jonathan Kelly added a comment - OK, I think it makes sense not to include in the 1.0 release. I have not even figured out yet why I can't get it to work with Spark on YARN. (Haven't gotten any time for that in the past couple of days and might not have any today either.) As for maintaining it once it's in Bigtop, I would be happy to do so.
          Hide
          cos Konstantin Boudnik added a comment -

          We are trying to hit ApacheCon time frame for the - very least - release candidate. So - ideally, I don't want to have any potentially disrupting changes getting in after first week of April.

          Show
          cos Konstantin Boudnik added a comment - We are trying to hit ApacheCon time frame for the - very least - release candidate. So - ideally, I don't want to have any potentially disrupting changes getting in after first week of April.
          Hide
          jongyoul Jongyoul Lee added a comment -

          I don't know when Bigtop 1.0 is released well, but I think Zeppelin has an early stage to be merged into Bigtop 1.0. Zeppelin developed very fast and become mature, even though Zeppelin's some option are changed after Jonathan Kelly submits this patch. I think Zeppelin 0.5 is the moderate integration point. For now, The version of Zeppelin is 0.5.0-SNAPSHOT. Finally, I'm willing to help to maintain Zeppelin in Bigtop, too.

          Show
          jongyoul Jongyoul Lee added a comment - I don't know when Bigtop 1.0 is released well, but I think Zeppelin has an early stage to be merged into Bigtop 1.0. Zeppelin developed very fast and become mature, even though Zeppelin's some option are changed after Jonathan Kelly submits this patch. I think Zeppelin 0.5 is the moderate integration point. For now, The version of Zeppelin is 0.5.0-SNAPSHOT. Finally, I'm willing to help to maintain Zeppelin in Bigtop, too.
          Hide
          rnowling RJ Nowling added a comment -

          Maybe we should Zeppelin as having "preview" or "alpha" status until we've had time to hammer it and make sure there are no obvious bugs in the packaging? If so, I see no reason to delay the release.

          Also, in keeping with BigTop's earlier decision to keep a maintainer file and drop unmaintained packages, we should ask who is willing to help maintain Zeppelin in BigTop and add them to the maintainers file. We shouldn't merge the patch until we know who will maintain it.

          YoungWoo Kim, Jongyoul Lee, Jonathan Kelly what do you think?

          Show
          rnowling RJ Nowling added a comment - Maybe we should Zeppelin as having "preview" or "alpha" status until we've had time to hammer it and make sure there are no obvious bugs in the packaging? If so, I see no reason to delay the release. Also, in keeping with BigTop's earlier decision to keep a maintainer file and drop unmaintained packages, we should ask who is willing to help maintain Zeppelin in BigTop and add them to the maintainers file. We shouldn't merge the patch until we know who will maintain it. YoungWoo Kim , Jongyoul Lee , Jonathan Kelly what do you think?
          Hide
          cos Konstantin Boudnik added a comment -

          Is this ready to be in 1.0? We didn't have Zeppelin on the release road-map, so shall we postpone it until the consequent release or do it right now?

          Show
          cos Konstantin Boudnik added a comment - Is this ready to be in 1.0? We didn't have Zeppelin on the release road-map, so shall we postpone it until the consequent release or do it right now?
          Hide
          warwithin YoungWoo Kim added a comment - - edited

          Thank you Jonathan Kelly. I would suggest that for making packaging and testing easy, it would be nice if the build pulls source from apache git repo or github because for now, Zeppelin do not have its official release tarball on archives. Please see BIGTOP-1527 or Bigtop's top-level README.

          diff --git a/bigtop.mk b/bigtop.mk
          index 8999533..32dd833 100644
          --- a/bigtop.mk
          +++ b/bigtop.mk
          @@ -285,6 +285,8 @@ ZEPPELIN_RELEASE_VERSION=1
           ZEPPELIN_TARBALL_DST=zeppelin-$(ZEPPELIN_BASE_VERSION).tar.gz
           ZEPPELIN_TARBALL_SRC=zeppelin-$(ZEPPELIN_BASE_VERSION).tgz
           ZEPPELIN_DOWNLOAD_PATH=/zeppelin/zeppelin-$(ZEPPELIN_BASE_VERSION)
          +ZEPPELIN_GIT_REPO=git://git.apache.org/incubator-zeppelin.git
          +ZEPPELIN_GIT_REF=master
           $(eval $(call PACKAGE,zeppelin,ZEPPELIN))
           
           # Phoenix
          
          
          Show
          warwithin YoungWoo Kim added a comment - - edited Thank you Jonathan Kelly . I would suggest that for making packaging and testing easy, it would be nice if the build pulls source from apache git repo or github because for now, Zeppelin do not have its official release tarball on archives. Please see BIGTOP-1527 or Bigtop's top-level README. diff --git a/bigtop.mk b/bigtop.mk index 8999533..32dd833 100644 --- a/bigtop.mk +++ b/bigtop.mk @@ -285,6 +285,8 @@ ZEPPELIN_RELEASE_VERSION=1 ZEPPELIN_TARBALL_DST=zeppelin-$(ZEPPELIN_BASE_VERSION).tar.gz ZEPPELIN_TARBALL_SRC=zeppelin-$(ZEPPELIN_BASE_VERSION).tgz ZEPPELIN_DOWNLOAD_PATH=/zeppelin/zeppelin-$(ZEPPELIN_BASE_VERSION) +ZEPPELIN_GIT_REPO=git: //git.apache.org/incubator-zeppelin.git +ZEPPELIN_GIT_REF=master $(eval $(call PACKAGE,zeppelin,ZEPPELIN)) # Phoenix
          Hide
          jonathak Jonathan Kelly added a comment -

          To be honest, I'm not sure why Bigtop apps have their conf directories set up like that; I just copied what other apps had. I mainly copied from the Spark app's common, deb and rpm files.

          As for Spark on YARN, I completely forgot to share that error log today, and I'm probably going to leave the office soon. I'm not sure yet if I'll have time to work on this tomorrow. If not, maybe Friday.

          Show
          jonathak Jonathan Kelly added a comment - To be honest, I'm not sure why Bigtop apps have their conf directories set up like that; I just copied what other apps had. I mainly copied from the Spark app's common, deb and rpm files. As for Spark on YARN, I completely forgot to share that error log today, and I'm probably going to leave the office soon. I'm not sure yet if I'll have time to work on this tomorrow. If not, maybe Friday.
          Hide
          jongyoul Jongyoul Lee added a comment - - edited

          Jonathan Kelly I have a question because I'm not familiar way on Bigtop. Is it a common way to configuration in /etc/{package_name}/conf? IMO, /etc/zeppelin is better than /etc/zeppelin/conf for configuration directory, or do you have any reason for that? Everythink except this looks good to me, and I'll check Yarn issue with deb

          Show
          jongyoul Jongyoul Lee added a comment - - edited Jonathan Kelly I have a question because I'm not familiar way on Bigtop. Is it a common way to configuration in /etc/{package_name}/conf? IMO, /etc/zeppelin is better than /etc/zeppelin/conf for configuration directory, or do you have any reason for that? Everythink except this looks good to me, and I'll check Yarn issue with deb
          Hide
          jonathak Jonathan Kelly added a comment - - edited

          I have tested this first patch with an RPM for AmazonLinux. I have gotten the Zeppelin Tutorial notebook to work for Spark Local and Spark Standalone but not yet for Spark on YARN.

          https://issues.apache.org/jira/secure/attachment/12707241/BIGTOP-1769.patch

          Show
          jonathak Jonathan Kelly added a comment - - edited I have tested this first patch with an RPM for AmazonLinux. I have gotten the Zeppelin Tutorial notebook to work for Spark Local and Spark Standalone but not yet for Spark on YARN. https://issues.apache.org/jira/secure/attachment/12707241/BIGTOP-1769.patch
          Hide
          jonathak Jonathan Kelly added a comment -

          Of course, I will share that tomorrow.

          Show
          jonathak Jonathan Kelly added a comment - Of course, I will share that tomorrow.
          Hide
          jongyoul Jongyoul Lee added a comment -

          Could you show me your error log and conf/interpreter.json?

          Show
          jongyoul Jongyoul Lee added a comment - Could you show me your error log and conf/interpreter.json?
          Hide
          jonathak Jonathan Kelly added a comment -

          Yes, I'm setting SPARK_YARN_JAR to /usr/lib/spark/lib/spark-assembly.jar, which is a symlink to the versioned file in the same directory.

          Show
          jonathak Jonathan Kelly added a comment - Yes, I'm setting SPARK_YARN_JAR to /usr/lib/spark/lib/spark-assembly.jar, which is a symlink to the versioned file in the same directory.
          Hide
          jongyoul Jongyoul Lee added a comment -

          Jonathan Kelly Do you set SPARK_YARN_JAR in your zeppelin-env.sh? This is a mandatory option for using Yarn because Zeppelin doesn't support something like spark-assembly*.jar by itself yet. I'm trying to patch Yarn without SPARK_YARN_JAR option.

          Show
          jongyoul Jongyoul Lee added a comment - Jonathan Kelly Do you set SPARK_YARN_JAR in your zeppelin-env.sh? This is a mandatory option for using Yarn because Zeppelin doesn't support something like spark-assembly*.jar by itself yet. I'm trying to patch Yarn without SPARK_YARN_JAR option.
          Hide
          jonathak Jonathan Kelly added a comment -

          Quick update: I've gotten a Zeppelin RPM working with Spark Local and Spark Standalone, but I'm having some trouble with Spark on YARN. Whether or not I figure that out first, hopefully I'll be able to submit my patch some time tomorrow. I'm just getting it reviewed internally for now.

          Show
          jonathak Jonathan Kelly added a comment - Quick update: I've gotten a Zeppelin RPM working with Spark Local and Spark Standalone, but I'm having some trouble with Spark on YARN. Whether or not I figure that out first, hopefully I'll be able to submit my patch some time tomorrow. I'm just getting it reviewed internally for now.
          Hide
          jonathak Jonathan Kelly added a comment -

          I figured out what was wrong with my Jenkins build. I had forgotten that I was running the build on a private host rather than a host in public EC2 (which normally should be OK but wasn't for this particular build), and I think I was running into a firewall issue because I can't seem to access github via the git:// protocol on that host. I instead configured the build to run on a public EC2 host, and I'm not running into the same issues. Now I'm just fixing up some minor issues with building the RPM, and I can probably submit a patch tomorrow.

          I still don't know what's wrong with the build on my Mac, but it's not critical for me to figure it out now.

          Show
          jonathak Jonathan Kelly added a comment - I figured out what was wrong with my Jenkins build. I had forgotten that I was running the build on a private host rather than a host in public EC2 (which normally should be OK but wasn't for this particular build), and I think I was running into a firewall issue because I can't seem to access github via the git:// protocol on that host. I instead configured the build to run on a public EC2 host, and I'm not running into the same issues. Now I'm just fixing up some minor issues with building the RPM, and I can probably submit a patch tomorrow. I still don't know what's wrong with the build on my Mac, but it's not critical for me to figure it out now.
          Hide
          kaiyzen Nate DAmico added a comment -

          Can you add package.json contents, or attach here, that its running off of, shouldn't have error installing q module via npm

          Show
          kaiyzen Nate DAmico added a comment - Can you add package.json contents, or attach here, that its running off of, shouldn't have error installing q module via npm
          Hide
          jonathak Jonathan Kelly added a comment -

          I found some docs that said that clearing the npm global and user caches (npm cache clear; sudo !!) could help. I didn't actually have NPM/Node.js installed (it looks like the build is just using embedded versions), so I installed them from MacPorts then cleared the caches and retried the build. Now I get a different error:

          [INFO] Running 'npm install --color=false' in /Volumes/Amazon/emr-workspace/src/Aws157Zeppelin/zeppelin-web
          [INFO] npm WARN package.json zeppelin-web@0.0.0 No repository field.
          [INFO] npm WARN unmet dependency /Volumes/Amazon/emr-workspace/src/Aws157Zeppelin/zeppelin-web/node_modules/grunt-contrib-watch/node_modules/gaze/node_modules/globule requires lodash@'~1.0.1' but will load
          [INFO] npm WARN unmet dependency /Volumes/Amazon/emr-workspace/src/Aws157Zeppelin/zeppelin-web/node_modules/grunt-contrib-watch/node_modules/lodash,
          [INFO] npm WARN unmet dependency which is version 2.4.1
          [INFO]
          [INFO] — frontend-maven-plugin:0.0.20:bower (bower install) @ zeppelin-web —
          [INFO] Running 'bower --allow-root install' in /Volumes/Amazon/emr-workspace/src/Aws157Zeppelin/zeppelin-web
          [INFO]
          [INFO] module.js:340
          [INFO] throw err;
          [INFO] ^
          [INFO] Error: Cannot find module 'q'
          [INFO] at Function.Module._resolveFilename (module.js:338:15)
          [INFO] at Function.Module._load (module.js:280:25)
          [INFO] at Module.require (module.js:364:17)
          [INFO] at require (module.js:380:17)
          [INFO] at Object.<anonymous> (/Volumes/Amazon/emr-workspace/src/Aws157Zeppelin/zeppelin-web/node_modules/bower/bin/bower:6:9)
          [INFO] at Module._compile (module.js:456:26)
          [INFO] at Object.Module._extensions..js (module.js:474:10)
          [INFO] at Module.load (module.js:356:32)
          [INFO] at Function.Module._load (module.js:312:12)
          [INFO] at Function.Module.runMain (module.js:497:10)

          I couldn't tell if the Zeppelin build is still using the embedded versions (which would probably be preferable) or the newly installed versions, so uninstalled NPM and Node.js to make sure it would use the embedded versions, and I still get the same error as above. So I think clearing the caches did something (whether it was good or bad, I can't tell, though I think the build did get slightly farther), but now I have a new problem.

          Show
          jonathak Jonathan Kelly added a comment - I found some docs that said that clearing the npm global and user caches (npm cache clear; sudo !!) could help. I didn't actually have NPM/Node.js installed (it looks like the build is just using embedded versions), so I installed them from MacPorts then cleared the caches and retried the build. Now I get a different error: [INFO] Running 'npm install --color=false' in /Volumes/Amazon/emr-workspace/src/Aws157Zeppelin/zeppelin-web [INFO] npm WARN package.json zeppelin-web@0.0.0 No repository field. [INFO] npm WARN unmet dependency /Volumes/Amazon/emr-workspace/src/Aws157Zeppelin/zeppelin-web/node_modules/grunt-contrib-watch/node_modules/gaze/node_modules/globule requires lodash@'~1.0.1' but will load [INFO] npm WARN unmet dependency /Volumes/Amazon/emr-workspace/src/Aws157Zeppelin/zeppelin-web/node_modules/grunt-contrib-watch/node_modules/lodash, [INFO] npm WARN unmet dependency which is version 2.4.1 [INFO] [INFO] — frontend-maven-plugin:0.0.20:bower (bower install) @ zeppelin-web — [INFO] Running 'bower --allow-root install' in /Volumes/Amazon/emr-workspace/src/Aws157Zeppelin/zeppelin-web [INFO] [INFO] module.js:340 [INFO] throw err; [INFO] ^ [INFO] Error: Cannot find module 'q' [INFO] at Function.Module._resolveFilename (module.js:338:15) [INFO] at Function.Module._load (module.js:280:25) [INFO] at Module.require (module.js:364:17) [INFO] at require (module.js:380:17) [INFO] at Object.<anonymous> (/Volumes/Amazon/emr-workspace/src/Aws157Zeppelin/zeppelin-web/node_modules/bower/bin/bower:6:9) [INFO] at Module._compile (module.js:456:26) [INFO] at Object.Module._extensions..js (module.js:474:10) [INFO] at Module.load (module.js:356:32) [INFO] at Function.Module._load (module.js:312:12) [INFO] at Function.Module.runMain (module.js:497:10) I couldn't tell if the Zeppelin build is still using the embedded versions (which would probably be preferable) or the newly installed versions, so uninstalled NPM and Node.js to make sure it would use the embedded versions, and I still get the same error as above. So I think clearing the caches did something (whether it was good or bad, I can't tell, though I think the build did get slightly farther), but now I have a new problem.
          Hide
          rnowling RJ Nowling added a comment -

          One of my colleagues ran into issues with NPM when running as root (in a VM or something). Don't have the exact details. Just thought I'd ask

          Show
          rnowling RJ Nowling added a comment - One of my colleagues ran into issues with NPM when running as root (in a VM or something). Don't have the exact details. Just thought I'd ask
          Hide
          jonathak Jonathan Kelly added a comment -

          Just to confirm, I should not use root, right? I'd never run a build of something, particularly something unfamiliar to me, as root. It wouldn't even be possible for me to run as root in our Jenkins build.

          Show
          jonathak Jonathan Kelly added a comment - Just to confirm, I should not use root, right? I'd never run a build of something, particularly something unfamiliar to me, as root. It wouldn't even be possible for me to run as root in our Jenkins build.
          Hide
          jonathak Jonathan Kelly added a comment -

          Unfortunately, no, it contains the same errors that appeared in the build output I pasted above.

          Show
          jonathak Jonathan Kelly added a comment - Unfortunately, no, it contains the same errors that appeared in the build output I pasted above.
          Hide
          rnowling RJ Nowling added a comment -

          Anything useful in /Volumes/Amazon/emr-workspace/src/Aws157Zeppelin/zeppelin-web/npm-debug.log ?

          Show
          rnowling RJ Nowling added a comment - Anything useful in /Volumes/Amazon/emr-workspace/src/Aws157Zeppelin/zeppelin-web/npm-debug.log ?
          Hide
          rnowling RJ Nowling added a comment -

          Unrelated but the spark and hadoop profiles will set the spark and hadoop versions.

          Show
          rnowling RJ Nowling added a comment - Unrelated but the spark and hadoop profiles will set the spark and hadoop versions.
          Hide
          jonathak Jonathan Kelly added a comment -

          I looked through more of the build output on my Mac and noticed that there was some more error output I should include:

          [INFO] npm http 304 https://registry.npmjs.org/retry/0.6.0
          [INFO]
          [INFO] Error making request.
          [INFO] Error: connect ECONNREFUSED
          [INFO] at errnoException (net.js:901:11)
          [INFO] at Object.afterConnect [as oncomplete] (net.js:892:19)
          [INFO]
          [INFO] Please report this full log at https://github.com/Medium/phantomjs
          [INFO] npm http 304 https://registry.npmjs.org/stringify-object
          [INFO] npm http 304 https://registry.npmjs.org/update-notifier/0.2.0
          [INFO] npm http 304 https://registry.npmjs.org/request-progress/0.3.0
          [INFO] npm http 304 https://registry.npmjs.org/tar-fs/0.5.2
          [INFO] npm http 304 https://registry.npmjs.org/osenv/0.1.0
          [INFO] npm http 304 https://registry.npmjs.org/tmp/0.0.23
          [INFO] npm ERR! weird error 1
          [INFO] npm ERR! Error: ENOENT, lstat '/var/folders/h8/hyj8y7xs1hq_q0kk3brwj5qw123t53/T/npm-34892-Tqgy7hof/1427140942695-0.7720878394320607/package/coverage/coverage.json'
          [INFO] npm ERR! If you need help, you may report this log at:
          [INFO] npm ERR! <http://github.com/isaacs/npm/issues>
          [INFO] npm ERR! or email it to:
          [INFO] npm ERR! <npm-@googlegroups.com>
          [INFO]
          [INFO] npm ERR! System Darwin 13.4.0
          [INFO] npm ERR! command "/Volumes/Amazon/emr-workspace/src/Aws157Zeppelin/zeppelin-web/node/node" "/Volumes/Amazon/emr-workspace/src/Aws157Zeppelin/zeppelin-web/node/npm/bin/npm-cli.js" "install" "--color=false"
          [INFO] npm ERR! cwd /Volumes/Amazon/emr-workspace/src/Aws157Zeppelin/zeppelin-web
          [INFO] npm ERR! node -v v0.10.18
          [INFO] npm ERR! npm -v 1.3.8
          [INFO] npm ERR! path /var/folders/h8/hyj8y7xs1hq_q0kk3brwj5qw123t53/T/npm-34892-Tqgy7hof/1427140942695-0.7720878394320607/package/coverage/coverage.json
          [INFO] npm ERR! fstream_path /var/folders/h8/hyj8y7xs1hq_q0kk3brwj5qw123t53/T/npm-34892-Tqgy7hof/1427140942695-0.7720878394320607/package/coverage/coverage.json
          [INFO] npm ERR! fstream_type File
          [INFO] npm ERR! fstream_class FileWriter
          [INFO] npm ERR! code ENOENT
          [INFO] npm ERR! errno 34
          [INFO] npm ERR! fstream_stack /Volumes/Amazon/emr-workspace/src/Aws157Zeppelin/zeppelin-web/node/npm/node_modules/fstream/lib/writer.js:284:26
          [INFO] npm ERR! fstream_stack Object.oncomplete (fs.js:107:15)
          [INFO] npm ERR! Error: ENOENT, lstat '/var/folders/h8/hyj8y7xs1hq_q0kk3brwj5qw123t53/T/npm-34892-Tqgy7hof/1427140939936-0.9542212507221848/package/test/fixtures/pgts.yaml'
          [INFO] npm ERR! If you need help, you may report this log at:
          [INFO] npm ERR! <http://github.com/isaacs/npm/issues>
          [INFO] npm ERR! or email it to:
          [INFO] npm ERR! <npm-@googlegroups.com>
          [INFO]
          [INFO] npm ERR! System Darwin 13.4.0
          [INFO] npm ERR! command "/Volumes/Amazon/emr-workspace/src/Aws157Zeppelin/zeppelin-web/node/node" "/Volumes/Amazon/emr-workspace/src/Aws157Zeppelin/zeppelin-web/node/npm/bin/npm-cli.js" "install" "--color=false"
          [INFO] npm ERR! cwd /Volumes/Amazon/emr-workspace/src/Aws157Zeppelin/zeppelin-web
          [INFO] npm ERR! node -v v0.10.18
          [INFO] npm ERR! npm -v 1.3.8
          [INFO] npm ERR! path /var/folders/h8/hyj8y7xs1hq_q0kk3brwj5qw123t53/T/npm-34892-Tqgy7hof/1427140939936-0.9542212507221848/package/test/fixtures/pgts.yaml
          [INFO] npm ERR! fstream_path /var/folders/h8/hyj8y7xs1hq_q0kk3brwj5qw123t53/T/npm-34892-Tqgy7hof/1427140939936-0.9542212507221848/package/test/fixtures/pgts.yaml
          [INFO] npm ERR! fstream_type File
          [INFO] npm ERR! fstream_class FileWriter
          [INFO] npm ERR! code ENOENT
          [INFO] npm ERR! errno 34
          [INFO] npm ERR! fstream_stack /Volumes/Amazon/emr-workspace/src/Aws157Zeppelin/zeppelin-web/node/npm/node_modules/fstream/lib/writer.js:284:26
          [INFO] npm ERR! fstream_stack Object.oncomplete (fs.js:107:15)
          [INFO] npm ERR!
          [INFO] npm ERR! Additional logging details can be found in:
          [INFO] npm ERR! /Volumes/Amazon/emr-workspace/src/Aws157Zeppelin/zeppelin-web/npm-debug.log
          [INFO] npm ERR! not ok code 0

          Show
          jonathak Jonathan Kelly added a comment - I looked through more of the build output on my Mac and noticed that there was some more error output I should include: [INFO] npm http 304 https://registry.npmjs.org/retry/0.6.0 [INFO] [INFO] Error making request. [INFO] Error: connect ECONNREFUSED [INFO] at errnoException (net.js:901:11) [INFO] at Object.afterConnect [as oncomplete] (net.js:892:19) [INFO] [INFO] Please report this full log at https://github.com/Medium/phantomjs [INFO] npm http 304 https://registry.npmjs.org/stringify-object [INFO] npm http 304 https://registry.npmjs.org/update-notifier/0.2.0 [INFO] npm http 304 https://registry.npmjs.org/request-progress/0.3.0 [INFO] npm http 304 https://registry.npmjs.org/tar-fs/0.5.2 [INFO] npm http 304 https://registry.npmjs.org/osenv/0.1.0 [INFO] npm http 304 https://registry.npmjs.org/tmp/0.0.23 [INFO] npm ERR! weird error 1 [INFO] npm ERR! Error: ENOENT, lstat '/var/folders/h8/hyj8y7xs1hq_q0kk3brwj5qw123t53/T/npm-34892-Tqgy7hof/1427140942695-0.7720878394320607/package/coverage/coverage.json' [INFO] npm ERR! If you need help, you may report this log at: [INFO] npm ERR! < http://github.com/isaacs/npm/issues > [INFO] npm ERR! or email it to: [INFO] npm ERR! <npm-@googlegroups.com> [INFO] [INFO] npm ERR! System Darwin 13.4.0 [INFO] npm ERR! command "/Volumes/Amazon/emr-workspace/src/Aws157Zeppelin/zeppelin-web/node/node" "/Volumes/Amazon/emr-workspace/src/Aws157Zeppelin/zeppelin-web/node/npm/bin/npm-cli.js" "install" "--color=false" [INFO] npm ERR! cwd /Volumes/Amazon/emr-workspace/src/Aws157Zeppelin/zeppelin-web [INFO] npm ERR! node -v v0.10.18 [INFO] npm ERR! npm -v 1.3.8 [INFO] npm ERR! path /var/folders/h8/hyj8y7xs1hq_q0kk3brwj5qw123t53/T/npm-34892-Tqgy7hof/1427140942695-0.7720878394320607/package/coverage/coverage.json [INFO] npm ERR! fstream_path /var/folders/h8/hyj8y7xs1hq_q0kk3brwj5qw123t53/T/npm-34892-Tqgy7hof/1427140942695-0.7720878394320607/package/coverage/coverage.json [INFO] npm ERR! fstream_type File [INFO] npm ERR! fstream_class FileWriter [INFO] npm ERR! code ENOENT [INFO] npm ERR! errno 34 [INFO] npm ERR! fstream_stack /Volumes/Amazon/emr-workspace/src/Aws157Zeppelin/zeppelin-web/node/npm/node_modules/fstream/lib/writer.js:284:26 [INFO] npm ERR! fstream_stack Object.oncomplete (fs.js:107:15) [INFO] npm ERR! Error: ENOENT, lstat '/var/folders/h8/hyj8y7xs1hq_q0kk3brwj5qw123t53/T/npm-34892-Tqgy7hof/1427140939936-0.9542212507221848/package/test/fixtures/pgts.yaml' [INFO] npm ERR! If you need help, you may report this log at: [INFO] npm ERR! < http://github.com/isaacs/npm/issues > [INFO] npm ERR! or email it to: [INFO] npm ERR! <npm-@googlegroups.com> [INFO] [INFO] npm ERR! System Darwin 13.4.0 [INFO] npm ERR! command "/Volumes/Amazon/emr-workspace/src/Aws157Zeppelin/zeppelin-web/node/node" "/Volumes/Amazon/emr-workspace/src/Aws157Zeppelin/zeppelin-web/node/npm/bin/npm-cli.js" "install" "--color=false" [INFO] npm ERR! cwd /Volumes/Amazon/emr-workspace/src/Aws157Zeppelin/zeppelin-web [INFO] npm ERR! node -v v0.10.18 [INFO] npm ERR! npm -v 1.3.8 [INFO] npm ERR! path /var/folders/h8/hyj8y7xs1hq_q0kk3brwj5qw123t53/T/npm-34892-Tqgy7hof/1427140939936-0.9542212507221848/package/test/fixtures/pgts.yaml [INFO] npm ERR! fstream_path /var/folders/h8/hyj8y7xs1hq_q0kk3brwj5qw123t53/T/npm-34892-Tqgy7hof/1427140939936-0.9542212507221848/package/test/fixtures/pgts.yaml [INFO] npm ERR! fstream_type File [INFO] npm ERR! fstream_class FileWriter [INFO] npm ERR! code ENOENT [INFO] npm ERR! errno 34 [INFO] npm ERR! fstream_stack /Volumes/Amazon/emr-workspace/src/Aws157Zeppelin/zeppelin-web/node/npm/node_modules/fstream/lib/writer.js:284:26 [INFO] npm ERR! fstream_stack Object.oncomplete (fs.js:107:15) [INFO] npm ERR! [INFO] npm ERR! Additional logging details can be found in: [INFO] npm ERR! /Volumes/Amazon/emr-workspace/src/Aws157Zeppelin/zeppelin-web/npm-debug.log [INFO] npm ERR! not ok code 0
          Hide
          jonathak Jonathan Kelly added a comment -

          Oops, sorry, I meant to include the build command. I'm running it as non-root.

          Here's the command I'm running: mvn -X clean install -Pspark-1.3 -Dspark.version=1.3.0 -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests

          Show
          jonathak Jonathan Kelly added a comment - Oops, sorry, I meant to include the build command. I'm running it as non-root. Here's the command I'm running: mvn -X clean install -Pspark-1.3 -Dspark.version=1.3.0 -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests
          Hide
          rnowling RJ Nowling added a comment - - edited

          Jonathan Kelly Are you running the build as a root user or non-root user? What is your build command?

          Show
          rnowling RJ Nowling added a comment - - edited Jonathan Kelly Are you running the build as a root user or non-root user? What is your build command?
          Hide
          jonathak Jonathan Kelly added a comment -

          I had already started working on this some time early last week, but I was running into some problems during the build, all during the "Zeppelin: web Application" submodule.

          On my Mac, I get the following error:

          [ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:0.0.20:npm (npm install) on project zeppelin-web: Failed to run task: 'npm install --color=false' failed. (error code 1) -> [Help 1]
          org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.github.eirslett:frontend-maven-plugin:0.0.20:npm (npm install) on project zeppelin-web: Failed to run task

          But on an EC2 node (with AmazonLinux), I get the following error:

          [INFO] bower angular-cookies#1.3.8 ECMDERR Failed to execute "git ls-remote --tags --heads git://github.com/angular/bower-angular-cookies.git", exit code of #128
          ...
          [ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:0.0.20:bower (bower install) on project zeppelin-web: Failed to run task: 'bower --allow-root install' failed. (error code 1) -> [Help 1]
          org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.github.eirslett:frontend-maven-plugin:0.0.20:bower (bower install) on project zeppelin-web: Failed to run task

          The failure is sometimes for a different package, but the failure is consistent. I haven't been able to get a build to succeed on EC2, even though it appears that it's just timing out caching dependencies.

          Show
          jonathak Jonathan Kelly added a comment - I had already started working on this some time early last week, but I was running into some problems during the build, all during the "Zeppelin: web Application" submodule. On my Mac, I get the following error: [ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:0.0.20:npm (npm install) on project zeppelin-web: Failed to run task: 'npm install --color=false' failed. (error code 1) -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.github.eirslett:frontend-maven-plugin:0.0.20:npm (npm install) on project zeppelin-web: Failed to run task But on an EC2 node (with AmazonLinux), I get the following error: [INFO] bower angular-cookies#1.3.8 ECMDERR Failed to execute "git ls-remote --tags --heads git://github.com/angular/bower-angular-cookies.git", exit code of #128 ... [ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:0.0.20:bower (bower install) on project zeppelin-web: Failed to run task: 'bower --allow-root install' failed. (error code 1) -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.github.eirslett:frontend-maven-plugin:0.0.20:bower (bower install) on project zeppelin-web: Failed to run task The failure is sometimes for a different package, but the failure is consistent. I haven't been able to get a build to succeed on EC2, even though it appears that it's just timing out caching dependencies.
          Hide
          warwithin YoungWoo Kim added a comment -

          Konstantin Boudnik, Please add Jongyoul Lee to contributor list. Thanks!

          Show
          warwithin YoungWoo Kim added a comment - Konstantin Boudnik , Please add Jongyoul Lee to contributor list. Thanks!
          Hide
          jongyoul Jongyoul Lee added a comment -

          RJ Nowling, YoungWoo Kim. I make a PR for integrating Zeppelin. Please assign it to me

          Show
          jongyoul Jongyoul Lee added a comment - RJ Nowling , YoungWoo Kim . I make a PR for integrating Zeppelin. Please assign it to me

            People

            • Assignee:
              jonathak Jonathan Kelly
              Reporter:
              jongyoul Jongyoul Lee
            • Votes:
              1 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development