Details

    • Type: New Feature
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.2.0
    • Component/s: deployment
    • Labels:
      None

      Description

      This is to add a Juju charm for the Apache Spark component, with an optional relation to the Apache Hadoop component.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user johnsca opened a pull request:

          https://github.com/apache/bigtop/pull/117

          BIGTOP-2477: Add Juju charm for spark component

          This is to add a Juju charm for the Apache Spark component, with an optional relation to the Apache Hadoop component.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/juju-solutions/bigtop spark

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/bigtop/pull/117.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #117


          commit 1c209df508afb7767f050e93979343f37ebba4a1
          Author: Cory Johns <johnsca@gmail.com>
          Date: 2016-05-24T21:49:54Z

          BIGTOP-2477: Add Juju charm for spark component


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user johnsca opened a pull request: https://github.com/apache/bigtop/pull/117 BIGTOP-2477 : Add Juju charm for spark component This is to add a Juju charm for the Apache Spark component, with an optional relation to the Apache Hadoop component. You can merge this pull request into a Git repository by running: $ git pull https://github.com/juju-solutions/bigtop spark Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/117.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #117 commit 1c209df508afb7767f050e93979343f37ebba4a1 Author: Cory Johns <johnsca@gmail.com> Date: 2016-05-24T21:49:54Z BIGTOP-2477 : Add Juju charm for spark component
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user petevg commented on the issue:

          https://github.com/apache/bigtop/pull/117

          When I run bundletester on this charm, on AWS, I get a failure during 02-smoke-test.py, and the sparkscale machines are blocked waiting for the relation to java.

          Will dig into it further – I suspect that it's just an issue w/ a test not setting up a relation.

          Here's the failure:

          ERROR: spark::02-smoke-test.py
          [/tmp/bundletester-oB8F_D/spark/tests/02-smoke-test.py exit 1]
          2016-08-01 17:35:51 Starting deployment of local.aws-test:miles
          2016-08-01 17:35:52 Deploying services...
          2016-08-01 17:37:00 Config specifies num units for subordinate: openjdk
          2016-08-01 17:37:01 Adding relations...
          2016-08-01 17:37:02 Adding relation spark:java <-> openjdk:java
          2016-08-01 17:39:30 Deployment complete in 219.51 seconds
          /home/petevg/.local/lib/python3.5/site-packages/path.py:1717: DeprecationWarning
          : path is deprecated. Use Path instead.
          warnings.warn(msg, DeprecationWarning)
          F
          ======================================================================
          FAIL: test_deploy (_main_.TestDeploy)
          ----------------------------------------------------------------------
          Traceback (most recent call last):
          File "/tmp/bundletester-oB8F_D/spark/tests/02-smoke-test.py", line 41, in test
          _deploy
          assert "completed" in output['status']
          AssertionError

          ----------------------------------------------------------------------
          Ran 1 test in 474.559s

          FAILED (failures=1)

          Show
          githubbot ASF GitHub Bot added a comment - Github user petevg commented on the issue: https://github.com/apache/bigtop/pull/117 When I run bundletester on this charm, on AWS, I get a failure during 02-smoke-test.py, and the sparkscale machines are blocked waiting for the relation to java. Will dig into it further – I suspect that it's just an issue w/ a test not setting up a relation. Here's the failure: ERROR: spark::02-smoke-test.py [/tmp/bundletester-oB8F_D/spark/tests/02-smoke-test.py exit 1] 2016-08-01 17:35:51 Starting deployment of local.aws-test:miles 2016-08-01 17:35:52 Deploying services... 2016-08-01 17:37:00 Config specifies num units for subordinate: openjdk 2016-08-01 17:37:01 Adding relations... 2016-08-01 17:37:02 Adding relation spark:java <-> openjdk:java 2016-08-01 17:39:30 Deployment complete in 219.51 seconds /home/petevg/.local/lib/python3.5/site-packages/path.py:1717: DeprecationWarning : path is deprecated. Use Path instead. warnings.warn(msg, DeprecationWarning) F ====================================================================== FAIL: test_deploy (_ main _.TestDeploy) ---------------------------------------------------------------------- Traceback (most recent call last): File "/tmp/bundletester-oB8F_D/spark/tests/02-smoke-test.py", line 41, in test _deploy assert "completed" in output ['status'] AssertionError ---------------------------------------------------------------------- Ran 1 test in 474.559s FAILED (failures=1)
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user petevg commented on the issue:

          https://github.com/apache/bigtop/pull/117

          I merged some fixes for the tests. They "HA" test still fails, but I believe that is related to outstanding issues w/ hadoop HA, which is one of the things that our team is currently working on.

          All other tests should pass when run w/ bundletester. Code looks good, too.

          Show
          githubbot ASF GitHub Bot added a comment - Github user petevg commented on the issue: https://github.com/apache/bigtop/pull/117 I merged some fixes for the tests. They "HA" test still fails, but I believe that is related to outstanding issues w/ hadoop HA, which is one of the things that our team is currently working on. All other tests should pass when run w/ bundletester. Code looks good, too.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user ktsakalozos commented on the issue:

          https://github.com/apache/bigtop/pull/117

          Thank you Pete.

          More work on HA + fixes are under review here: https://github.com/juju-solutions/bigtop/pull/37

          Show
          githubbot ASF GitHub Bot added a comment - Github user ktsakalozos commented on the issue: https://github.com/apache/bigtop/pull/117 Thank you Pete. More work on HA + fixes are under review here: https://github.com/juju-solutions/bigtop/pull/37
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user ktsakalozos commented on the issue:

          https://github.com/apache/bigtop/pull/117

          With the latest patch from https://github.com/juju-solutions/bigtop/pull/37 the HA tests are fixed. I am going to squash the commits in preparation for upstream merging

          Show
          githubbot ASF GitHub Bot added a comment - Github user ktsakalozos commented on the issue: https://github.com/apache/bigtop/pull/117 With the latest patch from https://github.com/juju-solutions/bigtop/pull/37 the HA tests are fixed. I am going to squash the commits in preparation for upstream merging
          Hide
          ktsakalozos Konstantinos Tsakalozos added a comment -

          Changes on puppet scripts to enable Spark HA

          Show
          ktsakalozos Konstantinos Tsakalozos added a comment - Changes on puppet scripts to enable Spark HA
          Hide
          kwmonroe Kevin W Monroe added a comment -

          I've tested this in 'standalone', 'HA', and 'yarn-client' mode using the following:

          https://jujucharms.com/spark/

          +1, LGTM.

          Show
          kwmonroe Kevin W Monroe added a comment - I've tested this in 'standalone', 'HA', and 'yarn-client' mode using the following: https://jujucharms.com/spark/ +1, LGTM.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/bigtop/pull/117

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/bigtop/pull/117
          Hide
          kwmonroe Kevin W Monroe added a comment -

          Charm built from this source is available at http://jujucharms.com/spark

          Show
          kwmonroe Kevin W Monroe added a comment - Charm built from this source is available at http://jujucharms.com/spark

            People

            • Assignee:
              johnsca Cory Johns
              Reporter:
              johnsca Cory Johns
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development