Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.2.0
    • Component/s: deployment
    • Labels:
      None

      Description

      Similar to #2616, we've made improvements to the spark-processing bundle that are ready for review and inclusion upstream:

      • refresh bundled charms, including the use of bigtop-zookeeper
      • reduce spark machine count (2 units instead of 3, which still allows HA without as much $$$ in cloud resources)
      • ganglia/rsyslog have been updated from trusty to xenial
      • add new machine constraints since aws now defaults to an 8gb root disk (we want 32g at a minimum)
      • run the bigtop smoke tests for spark/zookeeper
      • ensure the proper amulet test harness is included in the bundle tests

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/bigtop/pull/168

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/bigtop/pull/168
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user johnsca commented on the issue:

          https://github.com/apache/bigtop/pull/168

          This seems good to me. The tests are improved, and are passing for deployments on GCE and Azure, as evinced by http://cwr.vapour.ws/bundle_spark_processing/d3bc4a08ce3c4b42a64f19d9bef0baea/report.html

          :+1:

          Show
          githubbot ASF GitHub Bot added a comment - Github user johnsca commented on the issue: https://github.com/apache/bigtop/pull/168 This seems good to me. The tests are improved, and are passing for deployments on GCE and Azure, as evinced by http://cwr.vapour.ws/bundle_spark_processing/d3bc4a08ce3c4b42a64f19d9bef0baea/report.html :+1:
          Hide
          kwmonroe Kevin W Monroe added a comment -

          Bundle is available at:

          https://jujucharms.com/spark-processing/

          Daily tests of this bundle across various clouds are available here:

          http://data.vapour.ws/cwr-tests/results/bundle_spark_processing/index.html

          Show
          kwmonroe Kevin W Monroe added a comment - Bundle is available at: https://jujucharms.com/spark-processing/ Daily tests of this bundle across various clouds are available here: http://data.vapour.ws/cwr-tests/results/bundle_spark_processing/index.html
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user kwmonroe opened a pull request:

          https://github.com/apache/bigtop/pull/168

          BIGTOP-2617: refresh juju spark-processing bundle

          Similar to BIGTOP-2616, we've made improvements to the spark-processing bundle that are ready for review and inclusion upstream:

          • update readme for consistency with other bundles as well as calling out new sparkbench capabilities
          • refresh bundled charms, including the use of bigtop-zookeeper
          • reduce spark machine count (2 units instead of 3, which still allows HA without as much $$$ in cloud resources)
          • ganglia/rsyslog have been updated from trusty to xenial
          • add new machine constraints since aws now defaults to an 8gb root disk (we want 32g at a minimum)
          • run the bigtop smoke tests for spark/zookeeper
          • ensure the proper amulet test harness is included in the bundle tests

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/juju-solutions/bigtop bug/sp-cwr-fixes

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/bigtop/pull/168.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #168


          commit 6c26d315e74ffddfa61dad7a8aa9bb0251706bd0
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-10-28T15:14:48Z

          reduce machine count (2 sparks instead of 3); refresh ganglia/rsyslog; use bigtop zk; bump amulet timeout to an hour

          commit 56f03b60aa3f438aa848aa7a0087816f6cb274d5
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-10-28T16:58:09Z

          use charm smoke tests in our bundle test

          commit c02b8f8286952ad21a4ea9c1df63e035afea6cff
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-10-28T17:26:37Z

          bump spark revno (longer test timeout)

          commit 62e66da9f08e821c2a8931f1e9dda8361fe32b0d
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-10-28T21:11:00Z

          bump revno to pick up spark and zk test fixes

          commit 78c6d20b6b00eeab5912ae3f7b6279e8303488dc
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-10-28T21:12:17Z

          fix pgrep

          commit 1fd7e88230a4141d046db9713a71ec0bd89a96e6
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-10-28T21:40:36Z

          pickup zk-test fix from zk

          commit 32ac3e230c01ba0bfba373b76d6bc143f381b3b6
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-10-28T23:12:21Z

          refresh readme to be consistent with other bundles; add zk<->ganglia monitoring; add a working ganglia unit

          commit ffafa4f10cf0c15a1a910d8abcc7a6ef3f97e279
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-10-31T16:41:15Z

          update smoke failure message

          commit 37b8387948834a7ecb14f5b1cddcd84e572cd300
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-10-31T22:31:02Z

          bump spark revno for spark-bench fixes

          commit 412de035785fe0cc863714de7c985d23ef832ab1
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-11-02T04:23:17Z

          refresh spark charm and readme for new sparkbench suite

          commit ee85b893f47448bfcbca787fe7d81f8812d0647b
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-11-02T22:46:32Z

          charm revision bump

          • new ganglia charms report app version and fix '-broken' hook failures
          • new spark charm correctly runs sparkbench regardless of spark exe mode

          commit ac67e45672aa87b7db2d391cfc92201065067970
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-11-04T22:50:41Z

          bump spark rev to pick up testcase changes (be consistent with other bigtop charms) and app status reporting (no hard coded strings)

          commit d6230776c24a564bdfdb9e471fee2b963edeb160
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-11-05T22:37:57Z

          bump spark rev for updated test case and sparkbench composite is now seconds

          commit a72b3af219ecf21004c37d85f38c0872542ef242
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-11-07T17:49:08Z

          bump zk revno to pick up test rename and app version

          commit c6a83e7596aec9b2316027eb19b693a684593f16
          Author: Cory Johns <johnsca@gmail.com>
          Date: 2016-11-14T21:57:37Z

          spark-processing tests.yaml should include PPA for Amulet

          commit 73db79d1390b078ad1addd43f855adcb7aae5647
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-11-15T18:48:43Z

          bump rsyslog charm revs for lint fix

          commit 1e053bb5cfe2041af6efbd63efb3e667df78a053
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-11-23T01:36:35Z

          charm rev bump, new constraints

          • bump zk to latest (consistent bigtop_ prefix, readme tweaks)
          • aws defaults to 8gb root filesystem; bump this to 20gb

          commit 2afcc96b9eb3895af201f98f916a9bfa847949d3
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-11-24T19:48:14Z

          rev charms to pick up new swap handler

          commit 7b63bdd05982c3e94bef30e6ae0442372584e4c6
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-11-28T23:12:24Z

          workaround azure root-disk: https://bugs.launchpad.net/juju/+bug/1645408


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user kwmonroe opened a pull request: https://github.com/apache/bigtop/pull/168 BIGTOP-2617 : refresh juju spark-processing bundle Similar to BIGTOP-2616 , we've made improvements to the spark-processing bundle that are ready for review and inclusion upstream: update readme for consistency with other bundles as well as calling out new sparkbench capabilities refresh bundled charms, including the use of bigtop-zookeeper reduce spark machine count (2 units instead of 3, which still allows HA without as much $$$ in cloud resources) ganglia/rsyslog have been updated from trusty to xenial add new machine constraints since aws now defaults to an 8gb root disk (we want 32g at a minimum) run the bigtop smoke tests for spark/zookeeper ensure the proper amulet test harness is included in the bundle tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/juju-solutions/bigtop bug/sp-cwr-fixes Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/168.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #168 commit 6c26d315e74ffddfa61dad7a8aa9bb0251706bd0 Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-10-28T15:14:48Z reduce machine count (2 sparks instead of 3); refresh ganglia/rsyslog; use bigtop zk; bump amulet timeout to an hour commit 56f03b60aa3f438aa848aa7a0087816f6cb274d5 Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-10-28T16:58:09Z use charm smoke tests in our bundle test commit c02b8f8286952ad21a4ea9c1df63e035afea6cff Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-10-28T17:26:37Z bump spark revno (longer test timeout) commit 62e66da9f08e821c2a8931f1e9dda8361fe32b0d Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-10-28T21:11:00Z bump revno to pick up spark and zk test fixes commit 78c6d20b6b00eeab5912ae3f7b6279e8303488dc Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-10-28T21:12:17Z fix pgrep commit 1fd7e88230a4141d046db9713a71ec0bd89a96e6 Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-10-28T21:40:36Z pickup zk-test fix from zk commit 32ac3e230c01ba0bfba373b76d6bc143f381b3b6 Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-10-28T23:12:21Z refresh readme to be consistent with other bundles; add zk<->ganglia monitoring; add a working ganglia unit commit ffafa4f10cf0c15a1a910d8abcc7a6ef3f97e279 Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-10-31T16:41:15Z update smoke failure message commit 37b8387948834a7ecb14f5b1cddcd84e572cd300 Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-10-31T22:31:02Z bump spark revno for spark-bench fixes commit 412de035785fe0cc863714de7c985d23ef832ab1 Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-11-02T04:23:17Z refresh spark charm and readme for new sparkbench suite commit ee85b893f47448bfcbca787fe7d81f8812d0647b Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-11-02T22:46:32Z charm revision bump new ganglia charms report app version and fix '-broken' hook failures new spark charm correctly runs sparkbench regardless of spark exe mode commit ac67e45672aa87b7db2d391cfc92201065067970 Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-11-04T22:50:41Z bump spark rev to pick up testcase changes (be consistent with other bigtop charms) and app status reporting (no hard coded strings) commit d6230776c24a564bdfdb9e471fee2b963edeb160 Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-11-05T22:37:57Z bump spark rev for updated test case and sparkbench composite is now seconds commit a72b3af219ecf21004c37d85f38c0872542ef242 Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-11-07T17:49:08Z bump zk revno to pick up test rename and app version commit c6a83e7596aec9b2316027eb19b693a684593f16 Author: Cory Johns <johnsca@gmail.com> Date: 2016-11-14T21:57:37Z spark-processing tests.yaml should include PPA for Amulet commit 73db79d1390b078ad1addd43f855adcb7aae5647 Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-11-15T18:48:43Z bump rsyslog charm revs for lint fix commit 1e053bb5cfe2041af6efbd63efb3e667df78a053 Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-11-23T01:36:35Z charm rev bump, new constraints bump zk to latest (consistent bigtop_ prefix, readme tweaks) aws defaults to 8gb root filesystem; bump this to 20gb commit 2afcc96b9eb3895af201f98f916a9bfa847949d3 Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-11-24T19:48:14Z rev charms to pick up new swap handler commit 7b63bdd05982c3e94bef30e6ae0442372584e4c6 Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-11-28T23:12:24Z workaround azure root-disk: https://bugs.launchpad.net/juju/+bug/1645408

            People

            • Assignee:
              kwmonroe Kevin W Monroe
              Reporter:
              kwmonroe Kevin W Monroe
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development