Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-2797

zeppelin charm external role handling

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.1
    • Component/s: deployment
    • Labels:
      None

      Description

      While testing BIGTOP-2795, I discovered that the Zeppelin charm could easily foul up its Spark configuration. The charm was designed to be deployed by itself, with hadoop, with spark, or with hadoop+spark. There are a few problems:

      • Deploy zepp by itself; the first spark context job you process will fail. By default, the charm's spark configuration has an hdfs event log. No bueno if zepp is not related to hadoop.
      • Relate zepp to a standalone spark. Reconfigure that spark for yarn mode, and you'll break zepp. That's because Zepp was never related to hadoop, and therefore doesn't know how to deal with its local spark driver in 'yarn' mode.
      • Relate zepp to a standalone spark, and then remove that relation. You'll end up with zepp's spark driver in yarn-client mode. Yuck.

      As you can tell, Zeppelin really wants to be in yarn mode, but by golly, there's no real need for that. Let's handle the charm states (and state reactions) better.

        Issue Links

          Activity

          Hide
          kwmonroe Kevin W Monroe added a comment -

          Zepp charm looks good 4 ways:

          • standalone
          • +hadoop
          • +spark
          • +hadoop-spark

          PR merged; marking this Fixed.

          Show
          kwmonroe Kevin W Monroe added a comment - Zepp charm looks good 4 ways: standalone +hadoop +spark +hadoop-spark PR merged; marking this Fixed.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/bigtop/pull/230

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/bigtop/pull/230
          Hide
          kwmonroe Kevin W Monroe added a comment -

          zeppelin-20 is available for test:

          https://jujucharms.com/zeppelin/20

          Show
          kwmonroe Kevin W Monroe added a comment - zeppelin-20 is available for test: https://jujucharms.com/zeppelin/20
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user kwmonroe opened a pull request:

          https://github.com/apache/bigtop/pull/230

          BIGTOP-2797: zeppelin charm external role handling

          See [bigtop 2797](https://issues.apache.org/jira/browse/BIGTOP-2797) for details. Basically, the zeppelin charm should work by itself, with spark, with hadoop, and with hadoop+spark really well. Turns out it doesn't handle the internal spark config very well without hadoop being present.

          Also, our smoke test depends on the `requests` pip package, which has seen lots of recent changes. Pin the last known stable version.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/juju-solutions/bigtop bug/BIGTOP-2797/zeppelin-external-fixes

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/bigtop/pull/230.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #230


          commit cfdefb44e0fe80f021442a74b0e64ae10c9170f8
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2017-06-04T05:47:15Z

          handle external applications better

          • minor smoke test fix
          • port name change (websocket vs web)
          • always configure embedded spark in local mode with local event dir (otherwise we break when hdfs is not present)
          • remove unnecessary etc_env method
          • fix hive relation names (joined/ready vs connected/available)
          • block if we try to relate a non-standalone spark
          • reconfigure embedded spark appropriately if remote spark departs
          • pin requests to 2.14.2 (known breakage in 2.17.3)

          commit fafbf1bec1aa95fd249a64e237c248e13cb6831b
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2017-06-05T03:32:26Z

          run smoke test standalone first, then with an external spark

          commit 5c74d2cd0fe672c360661f133e6dcedf58939d4a
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2017-06-05T03:33:43Z

          detail deployment options and limitation of external spark

          commit 3b6fa202f970a032e77b6c771e5012088e347ccc
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2017-06-05T04:04:10Z

          make test executable


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user kwmonroe opened a pull request: https://github.com/apache/bigtop/pull/230 BIGTOP-2797 : zeppelin charm external role handling See [bigtop 2797] ( https://issues.apache.org/jira/browse/BIGTOP-2797 ) for details. Basically, the zeppelin charm should work by itself, with spark, with hadoop, and with hadoop+spark really well. Turns out it doesn't handle the internal spark config very well without hadoop being present. Also, our smoke test depends on the `requests` pip package, which has seen lots of recent changes. Pin the last known stable version. You can merge this pull request into a Git repository by running: $ git pull https://github.com/juju-solutions/bigtop bug/ BIGTOP-2797 /zeppelin-external-fixes Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/230.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #230 commit cfdefb44e0fe80f021442a74b0e64ae10c9170f8 Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2017-06-04T05:47:15Z handle external applications better minor smoke test fix port name change (websocket vs web) always configure embedded spark in local mode with local event dir (otherwise we break when hdfs is not present) remove unnecessary etc_env method fix hive relation names (joined/ready vs connected/available) block if we try to relate a non-standalone spark reconfigure embedded spark appropriately if remote spark departs pin requests to 2.14.2 (known breakage in 2.17.3) commit fafbf1bec1aa95fd249a64e237c248e13cb6831b Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2017-06-05T03:32:26Z run smoke test standalone first, then with an external spark commit 5c74d2cd0fe672c360661f133e6dcedf58939d4a Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2017-06-05T03:33:43Z detail deployment options and limitation of external spark commit 3b6fa202f970a032e77b6c771e5012088e347ccc Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2017-06-05T04:04:10Z make test executable

            People

            • Assignee:
              kwmonroe Kevin W Monroe
              Reporter:
              kwmonroe Kevin W Monroe
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development