Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-2765

fix roles logic for spark/zeppelin charms

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.1
    • Component/s: deployment
    • Labels:
      None

      Description

      The spark charm always includes the spark-worker role and also includes spark-master on the leader unit. This is not valid when spark is in yarn-* mode. There is no need for master/worker processes to be running when spark is in yarn mode.

      The zeppelin charm needs spark-datanucleus because the included spark bits are tightly coupled to hive. We manually apt install this package, but we should be using an appropriate role to leverage bigtop puppet just like we do for the spark charm. This only affects deployments without a separate spark unit, but that is a valid use case and we should have a nice UX for people that only deploy zeppelin.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user kwmonroe opened a pull request:

          https://github.com/apache/bigtop/pull/206

          BIGTOP-2765: update roles logic in spark/zeppelin

          Rework role logic for spark/zeppelin charms.

          • spark: only include master/worker when running in standalone mode
          • spark: include spark-yarn roles when in yarn-* mode
          • zeppelin: include spark-client and yarn roles to ensure the embedded spark is configured properly for standalone zeppelin deployments.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/juju-solutions/bigtop bug/BIGTOP-2765/spark-zepp-roles

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/bigtop/pull/206.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #206


          commit 65d12a96d98106e6ff6818cc60d2d7da2489ba1f
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2017-05-10T17:26:27Z

          BIGTOP-2765: update roles logic in spark/zeppelin


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user kwmonroe opened a pull request: https://github.com/apache/bigtop/pull/206 BIGTOP-2765 : update roles logic in spark/zeppelin Rework role logic for spark/zeppelin charms. spark: only include master/worker when running in standalone mode spark: include spark-yarn roles when in yarn-* mode zeppelin: include spark-client and yarn roles to ensure the embedded spark is configured properly for standalone zeppelin deployments. You can merge this pull request into a Git repository by running: $ git pull https://github.com/juju-solutions/bigtop bug/ BIGTOP-2765 /spark-zepp-roles Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/206.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #206 commit 65d12a96d98106e6ff6818cc60d2d7da2489ba1f Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2017-05-10T17:26:27Z BIGTOP-2765 : update roles logic in spark/zeppelin
          Hide
          kwmonroe Kevin W Monroe added a comment -

          Patch ready with linked PR. Testing now.

          Show
          kwmonroe Kevin W Monroe added a comment - Patch ready with linked PR. Testing now.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user johnsca commented on the issue:

          https://github.com/apache/bigtop/pull/206

          LGTM

          Show
          githubbot ASF GitHub Bot added a comment - Github user johnsca commented on the issue: https://github.com/apache/bigtop/pull/206 LGTM
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/bigtop/pull/206

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/bigtop/pull/206
          Hide
          kwmonroe Kevin W Monroe added a comment -

          Tests on aws/azure/gce looked good, so I'm closing this out.

          Show
          kwmonroe Kevin W Monroe added a comment - Tests on aws/azure/gce looked good, so I'm closing this out.

            People

            • Assignee:
              kwmonroe Kevin W Monroe
              Reporter:
              kwmonroe Kevin W Monroe
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 2h
                2h
                Remaining:
                Remaining Estimate - 2h
                2h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development