Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-2548

Refresh charms for Juju 2.0 and Xenial

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.2.0
    • Component/s: deployment
    • Labels:
      None

      Description

      With the recent Ubuntu long term support release (16.04 / Xenial) and in preparation for Juju 2.0, some of the core Hadoop charms need a refresh.

      Docs need to be updated and minor code changes need to be applied to ensure these charms will deploy on Xenial with Juju 2.0. Full list of changes:

      • update layer repo to upstream bigtop location
      • update metadata to inherit tags defined in bigtop base layer
      • update charm tests to use xenial base OS
      • alter plugin to correctly make java relation optional
      • use layer options to override NN and RM ports in puppet recipes
      • run bigtop smoke tests for RM (yarn) and Slave (hdfs + mapreduce)
      • run simple smoke tests for NN and plugin:
        • plugin lacks venv modules required to instantiate the Bigtop class, so it cannot call Bigtop.run_smoke_tests.
        • NN cannot successfully run the 'hdfs' smoke test. It fails on the distcp test, presumably because it lacks a component like the DN.
      • readme updates for consistent messaging and 2.0 instructions

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user kwmonroe opened a pull request:

          https://github.com/apache/bigtop/pull/148

          BIGTOP-2548: Refresh charms for Juju 2.0 and Xenial

          • update layer repo to upstream bigtop location
          • update metadata to inherit tags defined in bigtop base layer
          • update charm tests to use xenial base OS
          • alter plugin to correctly make java relation optional
          • use layer options to override NN and RM ports in puppet recipes
          • run bigtop smoke tests for RM (yarn) and Slave (hdfs + mapreduce)
          • run simple smoke tests for NN and plugin:
          • plugin lacks venv modules required to instantiate the Bigtop class, so it cannot call Bigtop.run_smoke_tests.
          • NN cannot successfully run the 'hdfs' smoke test. It fails on the distcp test, presumably because it lacks a component like the DN.
          • readme updates for consistent messaging and 2.0 instructions

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/juju-solutions/bigtop bug/BIGTOP-2548/xenial-charm-refresh

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/bigtop/pull/148.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #148



          Show
          githubbot ASF GitHub Bot added a comment - GitHub user kwmonroe opened a pull request: https://github.com/apache/bigtop/pull/148 BIGTOP-2548 : Refresh charms for Juju 2.0 and Xenial update layer repo to upstream bigtop location update metadata to inherit tags defined in bigtop base layer update charm tests to use xenial base OS alter plugin to correctly make java relation optional use layer options to override NN and RM ports in puppet recipes run bigtop smoke tests for RM (yarn) and Slave (hdfs + mapreduce) run simple smoke tests for NN and plugin: plugin lacks venv modules required to instantiate the Bigtop class, so it cannot call Bigtop.run_smoke_tests. NN cannot successfully run the 'hdfs' smoke test. It fails on the distcp test, presumably because it lacks a component like the DN. readme updates for consistent messaging and 2.0 instructions You can merge this pull request into a Git repository by running: $ git pull https://github.com/juju-solutions/bigtop bug/ BIGTOP-2548 /xenial-charm-refresh Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/148.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #148
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user johnsca commented on the issue:

          https://github.com/apache/bigtop/pull/148

          This is a pretty big PR, but it looks good. I was able to build and deploy the updated charms and run the smoke-tests. The README changes look good.

          The commits do need to be squashed, though.

          +1

          Show
          githubbot ASF GitHub Bot added a comment - Github user johnsca commented on the issue: https://github.com/apache/bigtop/pull/148 This is a pretty big PR, but it looks good. I was able to build and deploy the updated charms and run the smoke-tests. The README changes look good. The commits do need to be squashed, though. +1
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user petevg commented on the issue:

          https://github.com/apache/bigtop/pull/148

          I get an error when deploying the bundle, but it may just be me. Here's what I did:

          1) Edited bundle-local.yaml to fix the paths, as juju 2.0 doesn't like local:blah entries.
          2) Ran the following bash commands (I omitted some extraneous 'ls' and 'pwd' commands):
          ```
          2004 cd bigtop-packages/src/charm/hadoop/
          2007 rm -Rf ~/Code/charms/trusty/*
          2010 for i in `ls`; do charm build $i; done
          2012 cd ../../../../bigtop-deploy/juju/hadoop-processing/
          2014 juju deploy bundle-local.yaml
          ```
          3) I get the following error:

          ERROR cannot deploy bundle: cannot add relation between "openjdk" and "namenode": cannot add relation "openjdk:java namenode:java": principal and subordinate applications' series must match

          Show
          githubbot ASF GitHub Bot added a comment - Github user petevg commented on the issue: https://github.com/apache/bigtop/pull/148 I get an error when deploying the bundle, but it may just be me. Here's what I did: 1) Edited bundle-local.yaml to fix the paths, as juju 2.0 doesn't like local:blah entries. 2) Ran the following bash commands (I omitted some extraneous 'ls' and 'pwd' commands): ``` 2004 cd bigtop-packages/src/charm/hadoop/ 2007 rm -Rf ~/Code/charms/trusty/* 2010 for i in `ls`; do charm build $i; done 2012 cd ../../../../bigtop-deploy/juju/hadoop-processing/ 2014 juju deploy bundle-local.yaml ``` 3) I get the following error: ERROR cannot deploy bundle: cannot add relation between "openjdk" and "namenode": cannot add relation "openjdk:java namenode:java": principal and subordinate applications' series must match
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user johnsca commented on the issue:

          https://github.com/apache/bigtop/pull/148

          The charms must all be built and deployed as Xenial, including openjdk. However, we're also removing openjdk from the bundle, as it is now optional (for choosing a JDK other than openjdk, as openjdk is provided by default), so you can just remove that from the bundle entirely.

          The changes to the bundle should be addressed under https://issues.apache.org/jira/browse/BIGTOP-2550 once this is merged.

          Show
          githubbot ASF GitHub Bot added a comment - Github user johnsca commented on the issue: https://github.com/apache/bigtop/pull/148 The charms must all be built and deployed as Xenial, including openjdk. However, we're also removing openjdk from the bundle, as it is now optional (for choosing a JDK other than openjdk, as openjdk is provided by default), so you can just remove that from the bundle entirely. The changes to the bundle should be addressed under https://issues.apache.org/jira/browse/BIGTOP-2550 once this is merged.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/bigtop/pull/148

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/bigtop/pull/148
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kwmonroe commented on the issue:

          https://github.com/apache/bigtop/pull/148

          Thanks for the review guys. @johnsca you're right in that this was larger than expected, mostly due to the same changes (README/metadata/layer/smoke-test) going into each of the 4 hadoop charms (NN, RM, Slave, Plugin).

          In the future, I'll open a PR per charm to make the review more manageable.

          @petevg cory is correct in that testing these charms via the bundle won't be ready until BIGTOP-2550 is merged.

          Show
          githubbot ASF GitHub Bot added a comment - Github user kwmonroe commented on the issue: https://github.com/apache/bigtop/pull/148 Thanks for the review guys. @johnsca you're right in that this was larger than expected, mostly due to the same changes (README/metadata/layer/smoke-test) going into each of the 4 hadoop charms (NN, RM, Slave, Plugin). In the future, I'll open a PR per charm to make the review more manageable. @petevg cory is correct in that testing these charms via the bundle won't be ready until BIGTOP-2550 is merged.
          Show
          kwmonroe Kevin W Monroe added a comment - Charms built from these layers are available at: https://jujucharms.com/hadoop-namenode/ https://jujucharms.com/hadoop-resourcemanager/ https://jujucharms.com/hadoop-slave/ https://jujucharms.com/hadoop-plugin/
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user kwmonroe opened a pull request:

          https://github.com/apache/bigtop/pull/149

          Update juju hadoop bundle for Juju 2.0 and Xenial

          • Update README with xenial and juju2 instructions. Callout juju1 equivalent commands where applicable.
          • Colocate apps that can live together (eg: RM and NN) to cut down on required machine resources. Note this colocation in the readme.
          • Bump charm revisions to include newly built charms from BIGTOP-2548.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/juju-solutions/bigtop feature/BIGTOP-2550/bundle-to-xenial

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/bigtop/pull/149.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #149


          commit 3bb0d2fdbd185b18fe12617cf57e768817e2fc1a
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-09-08T19:21:27Z

          BIGTOP-2550: Update juju hadoop bundle for Juju 2.0 and Xenial

          commit 9c54752400e584a597cdfdaf8ce1d297d5510c23
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-10-10T16:38:50Z

          add free/trial resource limit workaround; add NN/RM utility instructions; minor cleanup

          commit 2305696cbd9a39cfdcb491762232ffd3c80965ce
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-10-10T18:30:30Z

          bump stable revnos (charm readme changes)

          commit 48dc4938e536e322e9acc75c34c0260b9553fcdd
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-10-10T22:05:03Z

          move network restricted info to deployment section; charm bump to pick up readme changes

          commit 5ff589990d7964f96a82e905598444f1f2497971
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-10-12T15:13:29Z

          format NN url


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user kwmonroe opened a pull request: https://github.com/apache/bigtop/pull/149 Update juju hadoop bundle for Juju 2.0 and Xenial Update README with xenial and juju2 instructions. Callout juju1 equivalent commands where applicable. Colocate apps that can live together (eg: RM and NN) to cut down on required machine resources. Note this colocation in the readme. Bump charm revisions to include newly built charms from BIGTOP-2548 . You can merge this pull request into a Git repository by running: $ git pull https://github.com/juju-solutions/bigtop feature/ BIGTOP-2550 /bundle-to-xenial Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/149.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #149 commit 3bb0d2fdbd185b18fe12617cf57e768817e2fc1a Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-09-08T19:21:27Z BIGTOP-2550 : Update juju hadoop bundle for Juju 2.0 and Xenial commit 9c54752400e584a597cdfdaf8ce1d297d5510c23 Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-10-10T16:38:50Z add free/trial resource limit workaround; add NN/RM utility instructions; minor cleanup commit 2305696cbd9a39cfdcb491762232ffd3c80965ce Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-10-10T18:30:30Z bump stable revnos (charm readme changes) commit 48dc4938e536e322e9acc75c34c0260b9553fcdd Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-10-10T22:05:03Z move network restricted info to deployment section; charm bump to pick up readme changes commit 5ff589990d7964f96a82e905598444f1f2497971 Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-10-12T15:13:29Z format NN url

            People

            • Assignee:
              kwmonroe Kevin W Monroe
              Reporter:
              kwmonroe Kevin W Monroe
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development