Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-2795

spark charm: fix sparkpi and rework start/stop logic

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2.1
    • Component/s: None
    • Labels:
      None

      Description

      The current spark charm has a couple problems:

      • The sample workload (delivered as ~/sparkpi.sh) hard codes the spark-examples.jar path, and it's incorrect for spark2.
      • Expose the spark worker ui port (8081) when appropriate.
      • We only need to open the master ui port (8080) if we're in standalone mode. There is no master process running in yarn or local modes.
      • We should only wait for master recovery if we have peers. It's important to wait for recovery before starting workers in a multi-node cluster, but for single-node, this is a waste of time.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/bigtop/pull/229

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/bigtop/pull/229
          Hide
          kwmonroe Kevin W Monroe added a comment -

          revision 45 of the spark charm contains these commits and is currently under test:

          https://jujucharms.com/spark/45

          Show
          kwmonroe Kevin W Monroe added a comment - revision 45 of the spark charm contains these commits and is currently under test: https://jujucharms.com/spark/45
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kwmonroe commented on the issue:

          https://github.com/apache/bigtop/pull/229

          During test, I found the `submit` action to be waaay too complex. I refactored it to use a simpler set of arguments, and then documented this in the readme.

          While I was in the readme, I noticed we didn't document the `*_memory` config options, so I added a section for those.

          Show
          githubbot ASF GitHub Bot added a comment - Github user kwmonroe commented on the issue: https://github.com/apache/bigtop/pull/229 During test, I found the `submit` action to be waaay too complex. I refactored it to use a simpler set of arguments, and then documented this in the readme. While I was in the readme, I noticed we didn't document the `*_memory` config options, so I added a section for those.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user kwmonroe opened a pull request:

          https://github.com/apache/bigtop/pull/229

          BIGTOP-2795: spark charm: fix sparkpi and rework start/stop logic

          See [bigtop-2795](https://issues.apache.org/jira/browse/BIGTOP-2795) for details.

          • add bundle descriptions to the hadoop-spark yaml files
          • expose the worker ui port
          • rework start/stop logic to handle ports and wait when appropriate
          • fix sparkpi.sh by not hard coding the spark-examples.jar path

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/juju-solutions/bigtop bug/BIGTOP-2795/logic-fixes

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/bigtop/pull/229.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #229


          commit 69e3e17574e67d38530fbe4beae8dd08e3dac6c2
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2017-06-01T20:57:30Z

          adjust bundle description

          commit e0e5d0cacdf41ae2e80c96f8b2251ec631da29d6
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2017-06-02T18:56:12Z

          add worker-ui port; rework start/stop to open ports and wait for services appropriately

          commit 71468acd0a47ae07f0488211df99a50309a2dabc
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2017-06-02T18:57:11Z

          sparkpi: find example jar dynamically and rework the output

          commit 3cd376fce4f0214b3ea9897213457c33cac9e725
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2017-06-02T19:01:24Z

          add bundle descriptions to dev/local bundle yamls


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user kwmonroe opened a pull request: https://github.com/apache/bigtop/pull/229 BIGTOP-2795 : spark charm: fix sparkpi and rework start/stop logic See [bigtop-2795] ( https://issues.apache.org/jira/browse/BIGTOP-2795 ) for details. add bundle descriptions to the hadoop-spark yaml files expose the worker ui port rework start/stop logic to handle ports and wait when appropriate fix sparkpi.sh by not hard coding the spark-examples.jar path You can merge this pull request into a Git repository by running: $ git pull https://github.com/juju-solutions/bigtop bug/ BIGTOP-2795 /logic-fixes Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/229.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #229 commit 69e3e17574e67d38530fbe4beae8dd08e3dac6c2 Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2017-06-01T20:57:30Z adjust bundle description commit e0e5d0cacdf41ae2e80c96f8b2251ec631da29d6 Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2017-06-02T18:56:12Z add worker-ui port; rework start/stop to open ports and wait for services appropriately commit 71468acd0a47ae07f0488211df99a50309a2dabc Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2017-06-02T18:57:11Z sparkpi: find example jar dynamically and rework the output commit 3cd376fce4f0214b3ea9897213457c33cac9e725 Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2017-06-02T19:01:24Z add bundle descriptions to dev/local bundle yamls

            People

            • Assignee:
              kwmonroe Kevin W Monroe
              Reporter:
              kwmonroe Kevin W Monroe
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development