Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.2.0
    • Component/s: deployment
    • Labels:

      Description

      This ticket is to track the progress of adding a charm that would deploy Zeppelin

        Issue Links

          Activity

          Hide
          kwmonroe Kevin W Monroe added a comment -

          Deployable Zeppelin charm available here:

          https://jujucharms.com/zeppelin/

          Show
          kwmonroe Kevin W Monroe added a comment - Deployable Zeppelin charm available here: https://jujucharms.com/zeppelin/
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/bigtop/pull/137

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/bigtop/pull/137
          Hide
          kwmonroe Kevin W Monroe added a comment -

          Thanks for addressing the review comments. This LGTM, verified on Azure and AWS.

          Show
          kwmonroe Kevin W Monroe added a comment - Thanks for addressing the review comments. This LGTM, verified on Azure and AWS.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user johnsca commented on a diff in the pull request:

          https://github.com/apache/bigtop/pull/137#discussion_r82406325

          — Diff: bigtop-packages/src/charm/zeppelin/layer-zeppelin/reactive/zeppelin.py —
          @@ -0,0 +1,158 @@
          +# Licensed to the Apache Software Foundation (ASF) under one or more
          +# contributor license agreements. See the NOTICE file distributed with
          +# this work for additional information regarding copyright ownership.
          +# The ASF licenses this file to You under the Apache License, Version 2.0
          +# (the "License"); you may not use this file except in compliance with
          +# the License. You may obtain a copy of the License at
          +#
          +# http://www.apache.org/licenses/LICENSE-2.0
          +#
          +# Unless required by applicable law or agreed to in writing, software
          +# distributed under the License is distributed on an "AS IS" BASIS,
          +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          +# See the License for the specific language governing permissions and
          +# limitations under the License.
          +
          +import hashlib
          +
          +from charms.reactive import when, when_not
          +from charms.reactive import is_state, set_state, remove_state
          +from charmhelpers.core import hookenv
          +from charms.layer.bigtop_zeppelin import Zeppelin
          +from charms.reactive.helpers import data_changed
          +
          +
          +@when('zeppelin.installed')
          +def update_status():
          + hadoop_joined = is_state('hadoop.joined')
          + hadoop_ready = is_state('hadoop.ready')
          + hive_joined = is_state('hive.connected')
          + hive_ready = is_state('hive.available')
          + spark_joined = is_state('spark.joined')
          + spark_ready = is_state('spark.ready')
          +
          + waiting_apps = []
          + ready_apps = []
          + # Check status of the hadoop plugin
          + if hadoop_joined and not hadoop_ready:
          + waiting_apps.append('hadoop')
          + elif hadoop_ready:
          + ready_apps.append('hadoop')
          +
          + # Check status of Hive
          + if hive_joined and not hive_ready:
          + waiting_apps.append('hive')
          + elif hive_ready:
          + ready_apps.append('hive')
          +
          + # Check status of Spark
          + if spark_joined and not spark_ready:
          + waiting_apps.append('spark')
          + elif spark_ready:
          + ready_apps.append('spark')
          +
          + # Set appropriate status based on the apps we checked above
          + if waiting_apps:
          + hookenv.status_set('waiting',
          + 'waiting for: {}'.format(' & '.join(waiting_apps)))
          + elif ready_apps:
          + hookenv.status_set('active',
          + 'ready with: {}'.format(' & '.join(ready_apps)))
          + else:
          + hookenv.status_set('active', 'ready')
          +
          +
          +@when('bigtop.available')
          +@when_not('zeppelin.installed')
          +def initial_setup():
          + hookenv.status_set('maintenance', 'installing zeppelin')
          + zeppelin = Zeppelin()
          + zeppelin.install()
          + zeppelin.setup_etc_env()
          + zeppelin.open_ports()
          + set_state('zeppelin.installed')
          + update_status()
          +
          +
          +@when('zeppelin.installed')
          +@when('hadoop.ready')
          +@when_not('zeppelin.hadoop.configured')
          +def configure_hadoop(hadoop):
          + zeppelin = Zeppelin()
          + zeppelin.configure_hadoop()
          + zeppelin.register_hadoop_notebooks()
          + set_state('zeppelin.hadoop.configured')
          +
          +
          +@when('zeppelin.installed')
          +@when_not('hadoop.ready')
          +@when('zeppelin.hadoop.configured')
          +def unconfigure_hadoop():
          + zeppelin = Zeppelin()
          + zeppelin.remove_hadoop_notebooks()
          + remove_state('zeppelin.hadoop.configured')
          +
          +
          +@when('zeppelin.installed', 'hive.ready')
          +def configure_hive(hive):
          + hive_ip = hive.get_private_ip()
          + hive_port = hive.get_port()
          + hive_url = 'jdbc:hive2://%s:%s' % (hive_ip, hive_port)
          + if data_changed('hive.connect', hive_url):
          + hookenv.status_set('maintenance', 'configuring hive')
          + zeppelin = Zeppelin()
          + zeppelin.configure_hive(hive_url)
          + set_state('zeppelin.hive.configured')
          + update_status()
          +
          +
          +@when('zeppelin.installed', 'zeppelin.hive.configured')
          +@when_not('hive.ready')
          +def unconfigure_hive():
          + hookenv.status_set('maintenance', 'removing hive relation')
          + zeppelin = Zeppelin()
          + zeppelin.configure_hive('jdbc:hive2://:')
          + remove_state('zeppelin.hive.configured')
          + update_status()
          +
          +
          +@when('zeppelin.installed', 'spark.ready')
          +def configure_spark(spark):
          + master_url = spark.get_master_url()
          + if data_changed('spark.master', master_url):
          + hookenv.status_set('maintenance', 'configuring spark')
          + zeppelin = Zeppelin()
          + zeppelin.configure_spark(master_url)
          + set_state('zeppelin.spark.configured')
          + update_status()
          +
          +
          +@when('zeppelin.installed', 'zeppelin.spark.configured')
          +@when_not('spark.ready')
          +def unconfigure_spark():
          + hookenv.status_set('maintenance', 'removing spark relation')
          + zeppelin = Zeppelin()
          + zeppelin.configure_spark(None)
          — End diff –

          Since the overrides are additive, removing one just leaves it as its current value, so we'll need to explicitly change it back to `yarn-client`.

          Show
          githubbot ASF GitHub Bot added a comment - Github user johnsca commented on a diff in the pull request: https://github.com/apache/bigtop/pull/137#discussion_r82406325 — Diff: bigtop-packages/src/charm/zeppelin/layer-zeppelin/reactive/zeppelin.py — @@ -0,0 +1,158 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import hashlib + +from charms.reactive import when, when_not +from charms.reactive import is_state, set_state, remove_state +from charmhelpers.core import hookenv +from charms.layer.bigtop_zeppelin import Zeppelin +from charms.reactive.helpers import data_changed + + +@when('zeppelin.installed') +def update_status(): + hadoop_joined = is_state('hadoop.joined') + hadoop_ready = is_state('hadoop.ready') + hive_joined = is_state('hive.connected') + hive_ready = is_state('hive.available') + spark_joined = is_state('spark.joined') + spark_ready = is_state('spark.ready') + + waiting_apps = [] + ready_apps = [] + # Check status of the hadoop plugin + if hadoop_joined and not hadoop_ready: + waiting_apps.append('hadoop') + elif hadoop_ready: + ready_apps.append('hadoop') + + # Check status of Hive + if hive_joined and not hive_ready: + waiting_apps.append('hive') + elif hive_ready: + ready_apps.append('hive') + + # Check status of Spark + if spark_joined and not spark_ready: + waiting_apps.append('spark') + elif spark_ready: + ready_apps.append('spark') + + # Set appropriate status based on the apps we checked above + if waiting_apps: + hookenv.status_set('waiting', + 'waiting for: {}'.format(' & '.join(waiting_apps))) + elif ready_apps: + hookenv.status_set('active', + 'ready with: {}'.format(' & '.join(ready_apps))) + else: + hookenv.status_set('active', 'ready') + + +@when('bigtop.available') +@when_not('zeppelin.installed') +def initial_setup(): + hookenv.status_set('maintenance', 'installing zeppelin') + zeppelin = Zeppelin() + zeppelin.install() + zeppelin.setup_etc_env() + zeppelin.open_ports() + set_state('zeppelin.installed') + update_status() + + +@when('zeppelin.installed') +@when('hadoop.ready') +@when_not('zeppelin.hadoop.configured') +def configure_hadoop(hadoop): + zeppelin = Zeppelin() + zeppelin.configure_hadoop() + zeppelin.register_hadoop_notebooks() + set_state('zeppelin.hadoop.configured') + + +@when('zeppelin.installed') +@when_not('hadoop.ready') +@when('zeppelin.hadoop.configured') +def unconfigure_hadoop(): + zeppelin = Zeppelin() + zeppelin.remove_hadoop_notebooks() + remove_state('zeppelin.hadoop.configured') + + +@when('zeppelin.installed', 'hive.ready') +def configure_hive(hive): + hive_ip = hive.get_private_ip() + hive_port = hive.get_port() + hive_url = 'jdbc:hive2://%s:%s' % (hive_ip, hive_port) + if data_changed('hive.connect', hive_url): + hookenv.status_set('maintenance', 'configuring hive') + zeppelin = Zeppelin() + zeppelin.configure_hive(hive_url) + set_state('zeppelin.hive.configured') + update_status() + + +@when('zeppelin.installed', 'zeppelin.hive.configured') +@when_not('hive.ready') +def unconfigure_hive(): + hookenv.status_set('maintenance', 'removing hive relation') + zeppelin = Zeppelin() + zeppelin.configure_hive('jdbc:hive2://:') + remove_state('zeppelin.hive.configured') + update_status() + + +@when('zeppelin.installed', 'spark.ready') +def configure_spark(spark): + master_url = spark.get_master_url() + if data_changed('spark.master', master_url): + hookenv.status_set('maintenance', 'configuring spark') + zeppelin = Zeppelin() + zeppelin.configure_spark(master_url) + set_state('zeppelin.spark.configured') + update_status() + + +@when('zeppelin.installed', 'zeppelin.spark.configured') +@when_not('spark.ready') +def unconfigure_spark(): + hookenv.status_set('maintenance', 'removing spark relation') + zeppelin = Zeppelin() + zeppelin.configure_spark(None) — End diff – Since the overrides are additive, removing one just leaves it as its current value, so we'll need to explicitly change it back to `yarn-client`.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user johnsca commented on a diff in the pull request:

          https://github.com/apache/bigtop/pull/137#discussion_r82397771

          — Diff: bigtop-packages/src/charm/zeppelin/layer-zeppelin/reactive/zeppelin.py —
          @@ -0,0 +1,158 @@
          +# Licensed to the Apache Software Foundation (ASF) under one or more
          +# contributor license agreements. See the NOTICE file distributed with
          +# this work for additional information regarding copyright ownership.
          +# The ASF licenses this file to You under the Apache License, Version 2.0
          +# (the "License"); you may not use this file except in compliance with
          +# the License. You may obtain a copy of the License at
          +#
          +# http://www.apache.org/licenses/LICENSE-2.0
          +#
          +# Unless required by applicable law or agreed to in writing, software
          +# distributed under the License is distributed on an "AS IS" BASIS,
          +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          +# See the License for the specific language governing permissions and
          +# limitations under the License.
          +
          +import hashlib
          +
          +from charms.reactive import when, when_not
          +from charms.reactive import is_state, set_state, remove_state
          +from charmhelpers.core import hookenv
          +from charms.layer.bigtop_zeppelin import Zeppelin
          +from charms.reactive.helpers import data_changed
          +
          +
          +@when('zeppelin.installed')
          +def update_status():
          + hadoop_joined = is_state('hadoop.joined')
          + hadoop_ready = is_state('hadoop.ready')
          + hive_joined = is_state('hive.connected')
          + hive_ready = is_state('hive.available')
          + spark_joined = is_state('spark.joined')
          + spark_ready = is_state('spark.ready')
          +
          + waiting_apps = []
          + ready_apps = []
          + # Check status of the hadoop plugin
          + if hadoop_joined and not hadoop_ready:
          + waiting_apps.append('hadoop')
          + elif hadoop_ready:
          + ready_apps.append('hadoop')
          +
          + # Check status of Hive
          + if hive_joined and not hive_ready:
          + waiting_apps.append('hive')
          + elif hive_ready:
          + ready_apps.append('hive')
          +
          + # Check status of Spark
          + if spark_joined and not spark_ready:
          + waiting_apps.append('spark')
          + elif spark_ready:
          + ready_apps.append('spark')
          +
          + # Set appropriate status based on the apps we checked above
          + if waiting_apps:
          + hookenv.status_set('waiting',
          + 'waiting for: {}'.format(' & '.join(waiting_apps)))
          + elif ready_apps:
          + hookenv.status_set('active',
          + 'ready with: {}'.format(' & '.join(ready_apps)))
          + else:
          + hookenv.status_set('active', 'ready')
          +
          +
          +@when('bigtop.available')
          +@when_not('zeppelin.installed')
          +def initial_setup():
          + hookenv.status_set('maintenance', 'installing zeppelin')
          + zeppelin = Zeppelin()
          + zeppelin.install()
          + zeppelin.setup_etc_env()
          + zeppelin.open_ports()
          + set_state('zeppelin.installed')
          + update_status()
          +
          +
          +@when('zeppelin.installed')
          +@when('hadoop.ready')
          +@when_not('zeppelin.hadoop.configured')
          +def configure_hadoop(hadoop):
          + zeppelin = Zeppelin()
          + zeppelin.configure_hadoop()
          + zeppelin.register_hadoop_notebooks()
          + set_state('zeppelin.hadoop.configured')
          +
          +
          +@when('zeppelin.installed')
          +@when_not('hadoop.ready')
          +@when('zeppelin.hadoop.configured')
          +def unconfigure_hadoop():
          + zeppelin = Zeppelin()
          + zeppelin.remove_hadoop_notebooks()
          + remove_state('zeppelin.hadoop.configured')
          +
          +
          +@when('zeppelin.installed', 'hive.ready')
          +def configure_hive(hive):
          + hive_ip = hive.get_private_ip()
          + hive_port = hive.get_port()
          + hive_url = 'jdbc:hive2://%s:%s' % (hive_ip, hive_port)
          + if data_changed('hive.connect', hive_url):
          + hookenv.status_set('maintenance', 'configuring hive')
          + zeppelin = Zeppelin()
          + zeppelin.configure_hive(hive_url)
          + set_state('zeppelin.hive.configured')
          + update_status()
          +
          +
          +@when('zeppelin.installed', 'zeppelin.hive.configured')
          +@when_not('hive.ready')
          +def unconfigure_hive():
          + hookenv.status_set('maintenance', 'removing hive relation')
          + zeppelin = Zeppelin()
          + zeppelin.configure_hive('jdbc:hive2://:')
          + remove_state('zeppelin.hive.configured')
          + update_status()
          +
          +
          +@when('zeppelin.installed', 'spark.ready')
          +def configure_spark(spark):
          + master_url = spark.get_master_url()
          + if data_changed('spark.master', master_url):
          + hookenv.status_set('maintenance', 'configuring spark')
          + zeppelin = Zeppelin()
          + zeppelin.configure_spark(master_url)
          + set_state('zeppelin.spark.configured')
          + update_status()
          +
          +
          +@when('zeppelin.installed', 'zeppelin.spark.configured')
          +@when_not('spark.ready')
          +def unconfigure_spark():
          + hookenv.status_set('maintenance', 'removing spark relation')
          + zeppelin = Zeppelin()
          + zeppelin.configure_spark(None)
          — End diff –

          If Spark isn't related to Zeppelin, there's a fairly good chance that it's not related to Hadoop, either, and thus it would fail in `yarn-client` mode.

          Perhaps a better solution would be to remove the override entirely and just let Bigtop fall back to its default naturally.

          Show
          githubbot ASF GitHub Bot added a comment - Github user johnsca commented on a diff in the pull request: https://github.com/apache/bigtop/pull/137#discussion_r82397771 — Diff: bigtop-packages/src/charm/zeppelin/layer-zeppelin/reactive/zeppelin.py — @@ -0,0 +1,158 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import hashlib + +from charms.reactive import when, when_not +from charms.reactive import is_state, set_state, remove_state +from charmhelpers.core import hookenv +from charms.layer.bigtop_zeppelin import Zeppelin +from charms.reactive.helpers import data_changed + + +@when('zeppelin.installed') +def update_status(): + hadoop_joined = is_state('hadoop.joined') + hadoop_ready = is_state('hadoop.ready') + hive_joined = is_state('hive.connected') + hive_ready = is_state('hive.available') + spark_joined = is_state('spark.joined') + spark_ready = is_state('spark.ready') + + waiting_apps = [] + ready_apps = [] + # Check status of the hadoop plugin + if hadoop_joined and not hadoop_ready: + waiting_apps.append('hadoop') + elif hadoop_ready: + ready_apps.append('hadoop') + + # Check status of Hive + if hive_joined and not hive_ready: + waiting_apps.append('hive') + elif hive_ready: + ready_apps.append('hive') + + # Check status of Spark + if spark_joined and not spark_ready: + waiting_apps.append('spark') + elif spark_ready: + ready_apps.append('spark') + + # Set appropriate status based on the apps we checked above + if waiting_apps: + hookenv.status_set('waiting', + 'waiting for: {}'.format(' & '.join(waiting_apps))) + elif ready_apps: + hookenv.status_set('active', + 'ready with: {}'.format(' & '.join(ready_apps))) + else: + hookenv.status_set('active', 'ready') + + +@when('bigtop.available') +@when_not('zeppelin.installed') +def initial_setup(): + hookenv.status_set('maintenance', 'installing zeppelin') + zeppelin = Zeppelin() + zeppelin.install() + zeppelin.setup_etc_env() + zeppelin.open_ports() + set_state('zeppelin.installed') + update_status() + + +@when('zeppelin.installed') +@when('hadoop.ready') +@when_not('zeppelin.hadoop.configured') +def configure_hadoop(hadoop): + zeppelin = Zeppelin() + zeppelin.configure_hadoop() + zeppelin.register_hadoop_notebooks() + set_state('zeppelin.hadoop.configured') + + +@when('zeppelin.installed') +@when_not('hadoop.ready') +@when('zeppelin.hadoop.configured') +def unconfigure_hadoop(): + zeppelin = Zeppelin() + zeppelin.remove_hadoop_notebooks() + remove_state('zeppelin.hadoop.configured') + + +@when('zeppelin.installed', 'hive.ready') +def configure_hive(hive): + hive_ip = hive.get_private_ip() + hive_port = hive.get_port() + hive_url = 'jdbc:hive2://%s:%s' % (hive_ip, hive_port) + if data_changed('hive.connect', hive_url): + hookenv.status_set('maintenance', 'configuring hive') + zeppelin = Zeppelin() + zeppelin.configure_hive(hive_url) + set_state('zeppelin.hive.configured') + update_status() + + +@when('zeppelin.installed', 'zeppelin.hive.configured') +@when_not('hive.ready') +def unconfigure_hive(): + hookenv.status_set('maintenance', 'removing hive relation') + zeppelin = Zeppelin() + zeppelin.configure_hive('jdbc:hive2://:') + remove_state('zeppelin.hive.configured') + update_status() + + +@when('zeppelin.installed', 'spark.ready') +def configure_spark(spark): + master_url = spark.get_master_url() + if data_changed('spark.master', master_url): + hookenv.status_set('maintenance', 'configuring spark') + zeppelin = Zeppelin() + zeppelin.configure_spark(master_url) + set_state('zeppelin.spark.configured') + update_status() + + +@when('zeppelin.installed', 'zeppelin.spark.configured') +@when_not('spark.ready') +def unconfigure_spark(): + hookenv.status_set('maintenance', 'removing spark relation') + zeppelin = Zeppelin() + zeppelin.configure_spark(None) — End diff – If Spark isn't related to Zeppelin, there's a fairly good chance that it's not related to Hadoop, either, and thus it would fail in `yarn-client` mode. Perhaps a better solution would be to remove the override entirely and just let Bigtop fall back to its default naturally.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kwmonroe commented on a diff in the pull request:

          https://github.com/apache/bigtop/pull/137#discussion_r81858734

          — Diff: bigtop-packages/src/charm/zeppelin/layer-zeppelin/README.md —
          @@ -0,0 +1,119 @@
          +<!--
          + Licensed to the Apache Software Foundation (ASF) under one or more
          + contributor license agreements. See the NOTICE file distributed with
          + this work for additional information regarding copyright ownership.
          + The ASF licenses this file to You under the Apache License, Version 2.0
          + (the "License"); you may not use this file except in compliance with
          + the License. You may obtain a copy of the License at
          +
          + http://www.apache.org/licenses/LICENSE-2.0
          +
          + Unless required by applicable law or agreed to in writing, software
          + distributed under the License is distributed on an "AS IS" BASIS,
          + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + See the License for the specific language governing permissions and
          + limitations under the License.
          +-->
          +## Overview
          +
          +Apache Zeppelin is a web-based notebook that enables interactive data analytics.
          +You can make beautiful data-driven, interactive, and collaborative documents
          +with SQL, Scala and more.
          +
          +As a Multi-purpose Notebook, Apache Zeppelin is the place for interactive:
          +
          + * Data Ingestion
          + * Data Discovery
          + * Data Analytics
          + * Data Visualization & Collaboration
          +
          +
          +## Usage
          +
          +This charm is intended to be deployed alongside one of the
          +[big data bundles](https://jujucharms.com/u/bigdata-charmers/#bundles).
          +For example:
          +
          + juju deploy hadoop-processing
          +
          +_*Note*: The above assumes Juju 2.0 or greater. If using an earlier version
          +of Juju, the syntax is `juju quickstart cs:bundle/hadoop-processing`._
          +
          +Now add Zeppelin and expose the web interface:
          +
          + juju deploy zeppelin
          + juju expose zeppelin
          +
          +Now relate Zeppelin to Java and the Hadoop cluster:
          +
          + juju add-relation zeppelin openjdk
          + juju add-relation zeppelin plugin
          +
          +Once deployment is complete, you will have an Apache Bigtop Hadoop platform
          +with Zeppelin available to create notebooks and start analyzing your data!
          +You can access the web interface at http://

          {zeppelin_unit_ip_address}

          :9080.
          +The ip address can be found by running`juju status zeppelin | grep public-address`.
          +
          +
          +## Status and Smoke Test
          +
          +The services provide extended status reporting to indicate when they are ready:
          +
          + juju status
          +
          +This is particularly useful when combined with `watch` to track the on-going
          +progress of the deployment:
          +
          + watch -n 0.5 juju status
          +
          +The message for each unit will provide information about that unit's state.
          +Once they all indicate that they are ready, you can perform a "smoke test"
          +to verify that Zeppelin is working as expected using the built-in `smoke-test`
          +action:
          +
          + juju run-action zeppelin/0 smoke-test
          +
          +_*Note*: The above assumes Juju 2.0 or greater. If using an earlier version
          +of Juju, the syntax is `juju action do zeppelin/0 smoke-test`._
          +
          +After a minute or so, you can check the results of the smoke test:
          +
          + juju show-action-status
          +
          +_*Note*: The above assumes Juju 2.0 or greater. If using an earlier version
          +of Juju, the syntax is `juju action status`._
          +
          +You will see `status: completed` if the smoke test was successful, or
          +`status: failed` if it was not. You can get more information on why it failed
          +via:
          +
          + juju show-action-output <action-id>
          +
          +_*Note*: The above assumes Juju 2.0 or greater. If using an earlier version
          +of Juju, the syntax is `juju action fetch <action-id>`._
          +
          +
          +## Limitations
          — End diff –

          Let's remove this section until https://github.com/juju-solutions/bigtop/issues/40 is fixed. Doesn't make any sense here since you can't change spark config on the bigtop-spark charm anyway.

          Show
          githubbot ASF GitHub Bot added a comment - Github user kwmonroe commented on a diff in the pull request: https://github.com/apache/bigtop/pull/137#discussion_r81858734 — Diff: bigtop-packages/src/charm/zeppelin/layer-zeppelin/README.md — @@ -0,0 +1,119 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +## Overview + +Apache Zeppelin is a web-based notebook that enables interactive data analytics. +You can make beautiful data-driven, interactive, and collaborative documents +with SQL, Scala and more. + +As a Multi-purpose Notebook, Apache Zeppelin is the place for interactive: + + * Data Ingestion + * Data Discovery + * Data Analytics + * Data Visualization & Collaboration + + +## Usage + +This charm is intended to be deployed alongside one of the + [big data bundles] ( https://jujucharms.com/u/bigdata-charmers/#bundles ). +For example: + + juju deploy hadoop-processing + +_* Note *: The above assumes Juju 2.0 or greater. If using an earlier version +of Juju, the syntax is `juju quickstart cs:bundle/hadoop-processing`._ + +Now add Zeppelin and expose the web interface: + + juju deploy zeppelin + juju expose zeppelin + +Now relate Zeppelin to Java and the Hadoop cluster: + + juju add-relation zeppelin openjdk + juju add-relation zeppelin plugin + +Once deployment is complete, you will have an Apache Bigtop Hadoop platform +with Zeppelin available to create notebooks and start analyzing your data! +You can access the web interface at http:// {zeppelin_unit_ip_address} :9080. +The ip address can be found by running`juju status zeppelin | grep public-address`. + + +## Status and Smoke Test + +The services provide extended status reporting to indicate when they are ready: + + juju status + +This is particularly useful when combined with `watch` to track the on-going +progress of the deployment: + + watch -n 0.5 juju status + +The message for each unit will provide information about that unit's state. +Once they all indicate that they are ready, you can perform a "smoke test" +to verify that Zeppelin is working as expected using the built-in `smoke-test` +action: + + juju run-action zeppelin/0 smoke-test + +_* Note *: The above assumes Juju 2.0 or greater. If using an earlier version +of Juju, the syntax is `juju action do zeppelin/0 smoke-test`._ + +After a minute or so, you can check the results of the smoke test: + + juju show-action-status + +_* Note *: The above assumes Juju 2.0 or greater. If using an earlier version +of Juju, the syntax is `juju action status`._ + +You will see `status: completed` if the smoke test was successful, or +`status: failed` if it was not. You can get more information on why it failed +via: + + juju show-action-output <action-id> + +_* Note *: The above assumes Juju 2.0 or greater. If using an earlier version +of Juju, the syntax is `juju action fetch <action-id>`._ + + +## Limitations — End diff – Let's remove this section until https://github.com/juju-solutions/bigtop/issues/40 is fixed. Doesn't make any sense here since you can't change spark config on the bigtop-spark charm anyway.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kwmonroe commented on a diff in the pull request:

          https://github.com/apache/bigtop/pull/137#discussion_r81864360

          — Diff: bigtop-packages/src/charm/zeppelin/layer-zeppelin/tests/01-zeppelin-smoke.py —
          @@ -0,0 +1,66 @@
          +#!/usr/bin/env python3
          +
          +# Licensed to the Apache Software Foundation (ASF) under one or more
          +# contributor license agreements. See the NOTICE file distributed with
          +# this work for additional information regarding copyright ownership.
          +# The ASF licenses this file to You under the Apache License, Version 2.0
          +# (the "License"); you may not use this file except in compliance with
          +# the License. You may obtain a copy of the License at
          +#
          +# http://www.apache.org/licenses/LICENSE-2.0
          +#
          +# Unless required by applicable law or agreed to in writing, software
          +# distributed under the License is distributed on an "AS IS" BASIS,
          +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          +# See the License for the specific language governing permissions and
          +# limitations under the License.
          +
          +import unittest
          +import amulet
          +
          +
          +class TestDeploy(unittest.TestCase):
          + """
          + Deployment and smoke test for the Apache Bigtop Zeppelin service.
          + """
          + @classmethod
          + def setUpClass(cls):
          — End diff –

          We can simplify this class by removing all references to 'openjdk' and 'java' below. Do it!

          Show
          githubbot ASF GitHub Bot added a comment - Github user kwmonroe commented on a diff in the pull request: https://github.com/apache/bigtop/pull/137#discussion_r81864360 — Diff: bigtop-packages/src/charm/zeppelin/layer-zeppelin/tests/01-zeppelin-smoke.py — @@ -0,0 +1,66 @@ +#!/usr/bin/env python3 + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import unittest +import amulet + + +class TestDeploy(unittest.TestCase): + """ + Deployment and smoke test for the Apache Bigtop Zeppelin service. + """ + @classmethod + def setUpClass(cls): — End diff – We can simplify this class by removing all references to 'openjdk' and 'java' below. Do it!
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kwmonroe commented on a diff in the pull request:

          https://github.com/apache/bigtop/pull/137#discussion_r81859067

          — Diff: bigtop-packages/src/charm/zeppelin/layer-zeppelin/README.md —
          @@ -0,0 +1,119 @@
          +<!--
          + Licensed to the Apache Software Foundation (ASF) under one or more
          + contributor license agreements. See the NOTICE file distributed with
          + this work for additional information regarding copyright ownership.
          + The ASF licenses this file to You under the Apache License, Version 2.0
          + (the "License"); you may not use this file except in compliance with
          + the License. You may obtain a copy of the License at
          +
          + http://www.apache.org/licenses/LICENSE-2.0
          +
          + Unless required by applicable law or agreed to in writing, software
          + distributed under the License is distributed on an "AS IS" BASIS,
          + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + See the License for the specific language governing permissions and
          + limitations under the License.
          +-->
          +## Overview
          +
          +Apache Zeppelin is a web-based notebook that enables interactive data analytics.
          +You can make beautiful data-driven, interactive, and collaborative documents
          +with SQL, Scala and more.
          +
          +As a Multi-purpose Notebook, Apache Zeppelin is the place for interactive:
          +
          + * Data Ingestion
          + * Data Discovery
          + * Data Analytics
          + * Data Visualization & Collaboration
          +
          +
          +## Usage
          +
          +This charm is intended to be deployed alongside one of the
          +[big data bundles](https://jujucharms.com/u/bigdata-charmers/#bundles).
          +For example:
          +
          + juju deploy hadoop-processing
          +
          +_*Note*: The above assumes Juju 2.0 or greater. If using an earlier version
          +of Juju, the syntax is `juju quickstart cs:bundle/hadoop-processing`._
          +
          +Now add Zeppelin and expose the web interface:
          +
          + juju deploy zeppelin
          + juju expose zeppelin
          +
          +Now relate Zeppelin to Java and the Hadoop cluster:
          +
          + juju add-relation zeppelin openjdk
          — End diff –

          Openjdk is optional now and won't be present in the Bigtop bundles... which will make this `add-relation` step fail. Yank it (and the Java comment above).

          Show
          githubbot ASF GitHub Bot added a comment - Github user kwmonroe commented on a diff in the pull request: https://github.com/apache/bigtop/pull/137#discussion_r81859067 — Diff: bigtop-packages/src/charm/zeppelin/layer-zeppelin/README.md — @@ -0,0 +1,119 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +## Overview + +Apache Zeppelin is a web-based notebook that enables interactive data analytics. +You can make beautiful data-driven, interactive, and collaborative documents +with SQL, Scala and more. + +As a Multi-purpose Notebook, Apache Zeppelin is the place for interactive: + + * Data Ingestion + * Data Discovery + * Data Analytics + * Data Visualization & Collaboration + + +## Usage + +This charm is intended to be deployed alongside one of the + [big data bundles] ( https://jujucharms.com/u/bigdata-charmers/#bundles ). +For example: + + juju deploy hadoop-processing + +_* Note *: The above assumes Juju 2.0 or greater. If using an earlier version +of Juju, the syntax is `juju quickstart cs:bundle/hadoop-processing`._ + +Now add Zeppelin and expose the web interface: + + juju deploy zeppelin + juju expose zeppelin + +Now relate Zeppelin to Java and the Hadoop cluster: + + juju add-relation zeppelin openjdk — End diff – Openjdk is optional now and won't be present in the Bigtop bundles... which will make this `add-relation` step fail. Yank it (and the Java comment above).
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kwmonroe commented on a diff in the pull request:

          https://github.com/apache/bigtop/pull/137#discussion_r81862021

          — Diff: bigtop-packages/src/charm/zeppelin/layer-zeppelin/lib/charms/layer/bigtop_zeppelin.py —
          @@ -0,0 +1,238 @@
          +# Licensed to the Apache Software Foundation (ASF) under one or more
          +# contributor license agreements. See the NOTICE file distributed with
          +# this work for additional information regarding copyright ownership.
          +# The ASF licenses this file to You under the Apache License, Version 2.0
          +# (the "License"); you may not use this file except in compliance with
          +# the License. You may obtain a copy of the License at
          +#
          +# http://www.apache.org/licenses/LICENSE-2.0
          +#
          +# Unless required by applicable law or agreed to in writing, software
          +# distributed under the License is distributed on an "AS IS" BASIS,
          +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          +# See the License for the specific language governing permissions and
          +# limitations under the License.
          +
          +import os
          +import json
          +import time
          +import socket
          +from urllib.parse import urljoin
          +
          +import requests
          +from path import Path
          +
          +from jujubigdata import utils
          +from charmhelpers.core import hookenv, host, unitdata
          +from charms import layer
          +from charms.layer.apache_bigtop_base import Bigtop
          +
          +
          +class Zeppelin(object):
          + """
          + This class manages Zeppelin.
          + """
          + def _init_(self):
          + self.dist_config = utils.DistConfig(
          + data=layer.options('apache-bigtop-base'))
          +
          + def _add_override(self, name, value):
          + unitdata.kv().update(

          { + name: value, + }

          , prefix='zeppelin.bigtop.overrides.')
          +
          + def install(self):
          + '''
          + Trigger the Bigtop puppet recipe that handles the Zepplin service.
          + '''
          + # Dirs are handled by the bigtop deb. No need to call out to
          + # dist_config to do that work.
          + self.dist_config.add_users()
          + self.trigger_bigtop()
          — End diff –

          If some poor soul changes our layer.yaml ports, they're in trouble. Let's honor them by actually setting the chosen port in an override before calling `self.trigger_bigtop()` here. Something like:

          ```python
          self._add_override('zeppelin::server::server_port', self.dist_config.port('zeppelin'))
          self._add_override('zeppelin::server::web_socket_port', self.dist_config.port('zeppelin_web'))
          ```

          Those should be the correct overrides, as defined in zeppelin's init.pp here:

          https://github.com/apache/bigtop/blob/master/bigtop-deploy/puppet/modules/zeppelin/manifests/init.pp#L26

          Show
          githubbot ASF GitHub Bot added a comment - Github user kwmonroe commented on a diff in the pull request: https://github.com/apache/bigtop/pull/137#discussion_r81862021 — Diff: bigtop-packages/src/charm/zeppelin/layer-zeppelin/lib/charms/layer/bigtop_zeppelin.py — @@ -0,0 +1,238 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os +import json +import time +import socket +from urllib.parse import urljoin + +import requests +from path import Path + +from jujubigdata import utils +from charmhelpers.core import hookenv, host, unitdata +from charms import layer +from charms.layer.apache_bigtop_base import Bigtop + + +class Zeppelin(object): + """ + This class manages Zeppelin. + """ + def _ init _(self): + self.dist_config = utils.DistConfig( + data=layer.options('apache-bigtop-base')) + + def _add_override(self, name, value): + unitdata.kv().update( { + name: value, + } , prefix='zeppelin.bigtop.overrides.') + + def install(self): + ''' + Trigger the Bigtop puppet recipe that handles the Zepplin service. + ''' + # Dirs are handled by the bigtop deb. No need to call out to + # dist_config to do that work. + self.dist_config.add_users() + self.trigger_bigtop() — End diff – If some poor soul changes our layer.yaml ports, they're in trouble. Let's honor them by actually setting the chosen port in an override before calling `self.trigger_bigtop()` here. Something like: ```python self._add_override('zeppelin::server::server_port', self.dist_config.port('zeppelin')) self._add_override('zeppelin::server::web_socket_port', self.dist_config.port('zeppelin_web')) ``` Those should be the correct overrides, as defined in zeppelin's init.pp here: https://github.com/apache/bigtop/blob/master/bigtop-deploy/puppet/modules/zeppelin/manifests/init.pp#L26
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kwmonroe commented on a diff in the pull request:

          https://github.com/apache/bigtop/pull/137#discussion_r81864099

          — Diff: bigtop-packages/src/charm/zeppelin/layer-zeppelin/reactive/zeppelin.py —
          @@ -0,0 +1,158 @@
          +# Licensed to the Apache Software Foundation (ASF) under one or more
          +# contributor license agreements. See the NOTICE file distributed with
          +# this work for additional information regarding copyright ownership.
          +# The ASF licenses this file to You under the Apache License, Version 2.0
          +# (the "License"); you may not use this file except in compliance with
          +# the License. You may obtain a copy of the License at
          +#
          +# http://www.apache.org/licenses/LICENSE-2.0
          +#
          +# Unless required by applicable law or agreed to in writing, software
          +# distributed under the License is distributed on an "AS IS" BASIS,
          +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          +# See the License for the specific language governing permissions and
          +# limitations under the License.
          +
          +import hashlib
          +
          +from charms.reactive import when, when_not
          +from charms.reactive import is_state, set_state, remove_state
          +from charmhelpers.core import hookenv
          +from charms.layer.bigtop_zeppelin import Zeppelin
          +from charms.reactive.helpers import data_changed
          +
          +
          +@when('zeppelin.installed')
          +def update_status():
          + hadoop_joined = is_state('hadoop.joined')
          + hadoop_ready = is_state('hadoop.ready')
          + hive_joined = is_state('hive.connected')
          + hive_ready = is_state('hive.available')
          + spark_joined = is_state('spark.joined')
          + spark_ready = is_state('spark.ready')
          +
          + waiting_apps = []
          + ready_apps = []
          + # Check status of the hadoop plugin
          + if hadoop_joined and not hadoop_ready:
          + waiting_apps.append('hadoop')
          + elif hadoop_ready:
          + ready_apps.append('hadoop')
          +
          + # Check status of Hive
          + if hive_joined and not hive_ready:
          + waiting_apps.append('hive')
          + elif hive_ready:
          + ready_apps.append('hive')
          +
          + # Check status of Spark
          + if spark_joined and not spark_ready:
          + waiting_apps.append('spark')
          + elif spark_ready:
          + ready_apps.append('spark')
          +
          + # Set appropriate status based on the apps we checked above
          + if waiting_apps:
          + hookenv.status_set('waiting',
          + 'waiting for: {}'.format(' & '.join(waiting_apps)))
          + elif ready_apps:
          + hookenv.status_set('active',
          + 'ready with: {}'.format(' & '.join(ready_apps)))
          + else:
          + hookenv.status_set('active', 'ready')
          +
          +
          +@when('bigtop.available')
          +@when_not('zeppelin.installed')
          +def initial_setup():
          + hookenv.status_set('maintenance', 'installing zeppelin')
          + zeppelin = Zeppelin()
          + zeppelin.install()
          + zeppelin.setup_etc_env()
          + zeppelin.open_ports()
          + set_state('zeppelin.installed')
          + update_status()
          +
          +
          +@when('zeppelin.installed')
          +@when('hadoop.ready')
          +@when_not('zeppelin.hadoop.configured')
          +def configure_hadoop(hadoop):
          + zeppelin = Zeppelin()
          + zeppelin.configure_hadoop()
          + zeppelin.register_hadoop_notebooks()
          + set_state('zeppelin.hadoop.configured')
          +
          +
          +@when('zeppelin.installed')
          +@when_not('hadoop.ready')
          +@when('zeppelin.hadoop.configured')
          +def unconfigure_hadoop():
          + zeppelin = Zeppelin()
          + zeppelin.remove_hadoop_notebooks()
          + remove_state('zeppelin.hadoop.configured')
          +
          +
          +@when('zeppelin.installed', 'hive.ready')
          +def configure_hive(hive):
          + hive_ip = hive.get_private_ip()
          + hive_port = hive.get_port()
          + hive_url = 'jdbc:hive2://%s:%s' % (hive_ip, hive_port)
          + if data_changed('hive.connect', hive_url):
          + hookenv.status_set('maintenance', 'configuring hive')
          + zeppelin = Zeppelin()
          + zeppelin.configure_hive(hive_url)
          + set_state('zeppelin.hive.configured')
          + update_status()
          +
          +
          +@when('zeppelin.installed', 'zeppelin.hive.configured')
          +@when_not('hive.ready')
          +def unconfigure_hive():
          + hookenv.status_set('maintenance', 'removing hive relation')
          + zeppelin = Zeppelin()
          + zeppelin.configure_hive('jdbc:hive2://:')
          + remove_state('zeppelin.hive.configured')
          + update_status()
          +
          +
          +@when('zeppelin.installed', 'spark.ready')
          +def configure_spark(spark):
          + master_url = spark.get_master_url()
          + if data_changed('spark.master', master_url):
          + hookenv.status_set('maintenance', 'configuring spark')
          + zeppelin = Zeppelin()
          + zeppelin.configure_spark(master_url)
          + set_state('zeppelin.spark.configured')
          + update_status()
          +
          +
          +@when('zeppelin.installed', 'zeppelin.spark.configured')
          +@when_not('spark.ready')
          +def unconfigure_spark():
          + hookenv.status_set('maintenance', 'removing spark relation')
          + zeppelin = Zeppelin()
          + zeppelin.configure_spark(None)
          — End diff –

          I think we should check for `hadoop.ready` here, and if that's true, use `yarn-client` as the spark master url instead of `None`. I'm pretty sure zeppelin can run it's internal spark interpreter in yarn-client mode without any external spark master.

          FWIW, bigtop defines `spark_master_url` as `yarn-client` by default:
          https://github.com/apache/bigtop/blob/master/bigtop-deploy/puppet/modules/zeppelin/manifests/init.pp#L25

          Show
          githubbot ASF GitHub Bot added a comment - Github user kwmonroe commented on a diff in the pull request: https://github.com/apache/bigtop/pull/137#discussion_r81864099 — Diff: bigtop-packages/src/charm/zeppelin/layer-zeppelin/reactive/zeppelin.py — @@ -0,0 +1,158 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import hashlib + +from charms.reactive import when, when_not +from charms.reactive import is_state, set_state, remove_state +from charmhelpers.core import hookenv +from charms.layer.bigtop_zeppelin import Zeppelin +from charms.reactive.helpers import data_changed + + +@when('zeppelin.installed') +def update_status(): + hadoop_joined = is_state('hadoop.joined') + hadoop_ready = is_state('hadoop.ready') + hive_joined = is_state('hive.connected') + hive_ready = is_state('hive.available') + spark_joined = is_state('spark.joined') + spark_ready = is_state('spark.ready') + + waiting_apps = [] + ready_apps = [] + # Check status of the hadoop plugin + if hadoop_joined and not hadoop_ready: + waiting_apps.append('hadoop') + elif hadoop_ready: + ready_apps.append('hadoop') + + # Check status of Hive + if hive_joined and not hive_ready: + waiting_apps.append('hive') + elif hive_ready: + ready_apps.append('hive') + + # Check status of Spark + if spark_joined and not spark_ready: + waiting_apps.append('spark') + elif spark_ready: + ready_apps.append('spark') + + # Set appropriate status based on the apps we checked above + if waiting_apps: + hookenv.status_set('waiting', + 'waiting for: {}'.format(' & '.join(waiting_apps))) + elif ready_apps: + hookenv.status_set('active', + 'ready with: {}'.format(' & '.join(ready_apps))) + else: + hookenv.status_set('active', 'ready') + + +@when('bigtop.available') +@when_not('zeppelin.installed') +def initial_setup(): + hookenv.status_set('maintenance', 'installing zeppelin') + zeppelin = Zeppelin() + zeppelin.install() + zeppelin.setup_etc_env() + zeppelin.open_ports() + set_state('zeppelin.installed') + update_status() + + +@when('zeppelin.installed') +@when('hadoop.ready') +@when_not('zeppelin.hadoop.configured') +def configure_hadoop(hadoop): + zeppelin = Zeppelin() + zeppelin.configure_hadoop() + zeppelin.register_hadoop_notebooks() + set_state('zeppelin.hadoop.configured') + + +@when('zeppelin.installed') +@when_not('hadoop.ready') +@when('zeppelin.hadoop.configured') +def unconfigure_hadoop(): + zeppelin = Zeppelin() + zeppelin.remove_hadoop_notebooks() + remove_state('zeppelin.hadoop.configured') + + +@when('zeppelin.installed', 'hive.ready') +def configure_hive(hive): + hive_ip = hive.get_private_ip() + hive_port = hive.get_port() + hive_url = 'jdbc:hive2://%s:%s' % (hive_ip, hive_port) + if data_changed('hive.connect', hive_url): + hookenv.status_set('maintenance', 'configuring hive') + zeppelin = Zeppelin() + zeppelin.configure_hive(hive_url) + set_state('zeppelin.hive.configured') + update_status() + + +@when('zeppelin.installed', 'zeppelin.hive.configured') +@when_not('hive.ready') +def unconfigure_hive(): + hookenv.status_set('maintenance', 'removing hive relation') + zeppelin = Zeppelin() + zeppelin.configure_hive('jdbc:hive2://:') + remove_state('zeppelin.hive.configured') + update_status() + + +@when('zeppelin.installed', 'spark.ready') +def configure_spark(spark): + master_url = spark.get_master_url() + if data_changed('spark.master', master_url): + hookenv.status_set('maintenance', 'configuring spark') + zeppelin = Zeppelin() + zeppelin.configure_spark(master_url) + set_state('zeppelin.spark.configured') + update_status() + + +@when('zeppelin.installed', 'zeppelin.spark.configured') +@when_not('spark.ready') +def unconfigure_spark(): + hookenv.status_set('maintenance', 'removing spark relation') + zeppelin = Zeppelin() + zeppelin.configure_spark(None) — End diff – I think we should check for `hadoop.ready` here, and if that's true, use `yarn-client` as the spark master url instead of `None`. I'm pretty sure zeppelin can run it's internal spark interpreter in yarn-client mode without any external spark master. FWIW, bigtop defines `spark_master_url` as `yarn-client` by default: https://github.com/apache/bigtop/blob/master/bigtop-deploy/puppet/modules/zeppelin/manifests/init.pp#L25
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user c0s commented on a diff in the pull request:

          https://github.com/apache/bigtop/pull/137#discussion_r74286278

          — Diff: bigtop-packages/src/charm/zeppelin/layer-zeppelin/LICENSE —
          @@ -0,0 +1,177 @@
          +
          — End diff –

          You don't need to include LICENSE file to each charm - it is already done at the top of the project.

          Show
          githubbot ASF GitHub Bot added a comment - Github user c0s commented on a diff in the pull request: https://github.com/apache/bigtop/pull/137#discussion_r74286278 — Diff: bigtop-packages/src/charm/zeppelin/layer-zeppelin/LICENSE — @@ -0,0 +1,177 @@ + — End diff – You don't need to include LICENSE file to each charm - it is already done at the top of the project.
          Hide
          ktsakalozos Konstantinos Tsakalozos added a comment -

          The work of Kevin is on the PR

          Show
          ktsakalozos Konstantinos Tsakalozos added a comment - The work of Kevin is on the PR
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user ktsakalozos opened a pull request:

          https://github.com/apache/bigtop/pull/137

          BIGTOP-2516: Add Zeppelin Charm

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/juju-solutions/bigtop BIGTOP-2516-Zeppelin

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/bigtop/pull/137.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #137


          commit 7650b439ec7b24d2948e668c4c08cf9aff28c9e1
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-06-07T02:21:51Z

          BIGTOP-2516: Add Zeppelin Charm


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user ktsakalozos opened a pull request: https://github.com/apache/bigtop/pull/137 BIGTOP-2516 : Add Zeppelin Charm You can merge this pull request into a Git repository by running: $ git pull https://github.com/juju-solutions/bigtop BIGTOP-2516 -Zeppelin Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/137.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #137 commit 7650b439ec7b24d2948e668c4c08cf9aff28c9e1 Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-06-07T02:21:51Z BIGTOP-2516 : Add Zeppelin Charm

            People

            • Assignee:
              kwmonroe Kevin W Monroe
              Reporter:
              ktsakalozos Konstantinos Tsakalozos
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development