Details

    • Type: Improvement
    • Status: Closed
    • Priority: Trivial
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.2.0
    • Component/s: deployment
    • Labels:
      None

      Description

      Building off the core hadoop and spark bundles, we've put together one that combines these two:

      http://jujucharms.com/hadoop-spark

      This deploys hadoop and spark in yarn-client mode. Tests are running now; so far, so good.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user kwmonroe opened a pull request:

          https://github.com/apache/bigtop/pull/166

          BIGTOP-2561: add juju bundle for hadoop-spark

          This combines hadoop-processing and spark-processing into a single bundle:

          https://jujucharms.com/hadoop-spark/

          Test results are available here across multiple clouds:

          http://data.vapour.ws/cwr-tests/results/bundle_hadoop_spark/index.html

          We are working through infrastructure issues atm (hence a lot of yellow triangles), but we haven't seen any failures with the charms included in this bundle for the last couple weeks. Hence I'm putting it up for review / inclusion upstream.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/juju-solutions/bigtop feature/hadoop-spark-bundle

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/bigtop/pull/166.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #166


          commit 35b8d13c198a98296f0290e340e7bc93a7e1b62d
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-09-22T01:46:24Z

          initial hadoop-spark bundle

          commit d1387eda17b3d8cb326e5d0d12500a60ac66d04a
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-09-22T23:20:51Z

          colocate bundle applications

          To save resources, colocate some of the applications when it makes sense.
          Put the namenode and resourcemgr on the same unit; spark and zeppelin;
          ganglia and rsyslog.

          Update readme formatting and callout these colocated apps. Also tweak
          the restricted network section and add a section for connecting to zepp.

          Note, machine spec in the bundle yaml start with machine '1' instead of
          '0'. This is because someone may deploy this on juju-1.x, where machine
          '0' is the bootstrap node.

          commit b52a7c5a7020ffaec25ca3f240768eaf3b4c1cac
          Author: Konstantinos Tsakalozos <kos.tsakalozos@canonical.com>
          Date: 2016-10-24T09:37:24Z

          Be explicit on series for CWR to deploy the right series

          commit 040bab24a58d0b4eda05a8ab215342a1f21ff9a7
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-10-29T02:19:05Z

          refresh charms

          commit a29bac836b7d39adabe5d3c835f3d00e28b75f8a
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-10-29T05:03:24Z

          refresh hadoop-spark bundle

          • update readme to be consistent with other bigtop-based bundles
          • bump all charms to latest revisions
          • update amulet test to call charm smoke tests and work around machine spec

          commit ef3d70542babecd69baa0618e39388854968fda8
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-10-31T16:45:08Z

          skip slave smoke tests and update smoke failure message

          commit 506ba320764323f4ffb294bce4834f044626e799
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-10-31T22:29:15Z

          bump spark revno for spark-bench fixes

          commit 4c0f812d736b8d4c31bfce68b450d8b77adf79a0
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-11-02T04:25:58Z

          refresh spark charm and readme for new sparkbench suite

          commit ff19096dbb9bf3fb21297fcc0a258d98453ea7cc
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-11-03T04:33:14Z

          charm revision bump

          • new ganglia charms report app version and fix '-broken' hook failures
          • new spark charm correctly runs sparkbench regardless of spark exe mode

          commit 5a961a6fb15f403f80356de62b3acdaa7aa2b10c
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-11-05T22:39:41Z

          bump spark rev for updated test case and sparkbench composite is now seconds

          commit 71c5731208c637bc2cb0ed2a44179c25479dc5e2
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-11-07T17:53:37Z

          bump zk revno to pick up test rename and app version

          commit 745deec545226a5c9d83d9d749101bf227b4dec6
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-11-15T21:12:22Z

          bump rsyslog for lint and zk for an amulet test enhancement (ease local testing)

          commit 638221be832236641eef3ceef0a9b46e11121077
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-11-22T21:34:35Z

          add ppa, charm rev bump, new constraints

          • ensure tests use juju/stable ppa
          • bump zk to latest (consistent bigtop_ prefix, readme tweaks)
          • bump core hadoop charms to latest (new swap handler, detect failed service
            start, add spark user/group to nn/rm)
          • aws defaults to 8gb root filesystem; bump this to 20gb

          commit 34ffbe4bc576ba3a330e8c1da6dd08a74f3a392a
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-11-24T19:49:56Z

          rev charms to pick up new swap handler

          commit 3fc9934e7723e1ab8383d1a96c093f86d546178b
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-11-28T23:16:11Z

          workaround azure root-disk: https://bugs.launchpad.net/juju/+bug/1645408


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user kwmonroe opened a pull request: https://github.com/apache/bigtop/pull/166 BIGTOP-2561 : add juju bundle for hadoop-spark This combines hadoop-processing and spark-processing into a single bundle: https://jujucharms.com/hadoop-spark/ Test results are available here across multiple clouds: http://data.vapour.ws/cwr-tests/results/bundle_hadoop_spark/index.html We are working through infrastructure issues atm (hence a lot of yellow triangles), but we haven't seen any failures with the charms included in this bundle for the last couple weeks. Hence I'm putting it up for review / inclusion upstream. You can merge this pull request into a Git repository by running: $ git pull https://github.com/juju-solutions/bigtop feature/hadoop-spark-bundle Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/166.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #166 commit 35b8d13c198a98296f0290e340e7bc93a7e1b62d Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-09-22T01:46:24Z initial hadoop-spark bundle commit d1387eda17b3d8cb326e5d0d12500a60ac66d04a Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-09-22T23:20:51Z colocate bundle applications To save resources, colocate some of the applications when it makes sense. Put the namenode and resourcemgr on the same unit; spark and zeppelin; ganglia and rsyslog. Update readme formatting and callout these colocated apps. Also tweak the restricted network section and add a section for connecting to zepp. Note, machine spec in the bundle yaml start with machine '1' instead of '0'. This is because someone may deploy this on juju-1.x, where machine '0' is the bootstrap node. commit b52a7c5a7020ffaec25ca3f240768eaf3b4c1cac Author: Konstantinos Tsakalozos <kos.tsakalozos@canonical.com> Date: 2016-10-24T09:37:24Z Be explicit on series for CWR to deploy the right series commit 040bab24a58d0b4eda05a8ab215342a1f21ff9a7 Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-10-29T02:19:05Z refresh charms commit a29bac836b7d39adabe5d3c835f3d00e28b75f8a Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-10-29T05:03:24Z refresh hadoop-spark bundle update readme to be consistent with other bigtop-based bundles bump all charms to latest revisions update amulet test to call charm smoke tests and work around machine spec commit ef3d70542babecd69baa0618e39388854968fda8 Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-10-31T16:45:08Z skip slave smoke tests and update smoke failure message commit 506ba320764323f4ffb294bce4834f044626e799 Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-10-31T22:29:15Z bump spark revno for spark-bench fixes commit 4c0f812d736b8d4c31bfce68b450d8b77adf79a0 Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-11-02T04:25:58Z refresh spark charm and readme for new sparkbench suite commit ff19096dbb9bf3fb21297fcc0a258d98453ea7cc Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-11-03T04:33:14Z charm revision bump new ganglia charms report app version and fix '-broken' hook failures new spark charm correctly runs sparkbench regardless of spark exe mode commit 5a961a6fb15f403f80356de62b3acdaa7aa2b10c Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-11-05T22:39:41Z bump spark rev for updated test case and sparkbench composite is now seconds commit 71c5731208c637bc2cb0ed2a44179c25479dc5e2 Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-11-07T17:53:37Z bump zk revno to pick up test rename and app version commit 745deec545226a5c9d83d9d749101bf227b4dec6 Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-11-15T21:12:22Z bump rsyslog for lint and zk for an amulet test enhancement (ease local testing) commit 638221be832236641eef3ceef0a9b46e11121077 Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-11-22T21:34:35Z add ppa, charm rev bump, new constraints ensure tests use juju/stable ppa bump zk to latest (consistent bigtop_ prefix, readme tweaks) bump core hadoop charms to latest (new swap handler, detect failed service start, add spark user/group to nn/rm) aws defaults to 8gb root filesystem; bump this to 20gb commit 34ffbe4bc576ba3a330e8c1da6dd08a74f3a392a Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-11-24T19:49:56Z rev charms to pick up new swap handler commit 3fc9934e7723e1ab8383d1a96c093f86d546178b Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-11-28T23:16:11Z workaround azure root-disk: https://bugs.launchpad.net/juju/+bug/1645408
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user petevg commented on the issue:

          https://github.com/apache/bigtop/pull/166

          LGTM/+1 (I was just able to deploy this bundle to aws without any hitches, too)

          Show
          githubbot ASF GitHub Bot added a comment - Github user petevg commented on the issue: https://github.com/apache/bigtop/pull/166 LGTM/+1 (I was just able to deploy this bundle to aws without any hitches, too)
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/bigtop/pull/166

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/bigtop/pull/166
          Hide
          kwmonroe Kevin W Monroe added a comment -

          This bundle is available at:

          http://jujucharms.com/hadoop-spark

          It's also part of the daily cloud-weather-report runs with results available here:

          http://data.vapour.ws/cwr-tests/results/bundle_hadoop_spark/index.html

          Show
          kwmonroe Kevin W Monroe added a comment - This bundle is available at: http://jujucharms.com/hadoop-spark It's also part of the daily cloud-weather-report runs with results available here: http://data.vapour.ws/cwr-tests/results/bundle_hadoop_spark/index.html

            People

            • Assignee:
              kwmonroe Kevin W Monroe
              Reporter:
              kwmonroe Kevin W Monroe
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development