Details

    • Type: New Feature
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.2.0
    • Component/s: deployment
    • Labels:

      Description

      Adding a Juju charm to deploy Mahout.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user ktsakalozos opened a pull request:

          https://github.com/apache/bigtop/pull/123

          BIGTOP-2483: Add Mahout Charm

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/juju-solutions/bigtop mahout

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/bigtop/pull/123.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #123


          commit f61ab1831caaed7f0da68a37a07076c480f411c3
          Author: Konstantinos Tsakalozos <konstantinos.tsakalozos@canonical.com>
          Date: 2016-06-07T14:01:49Z

          BIGTOP-2483: Add Mahout Charm


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user ktsakalozos opened a pull request: https://github.com/apache/bigtop/pull/123 BIGTOP-2483 : Add Mahout Charm You can merge this pull request into a Git repository by running: $ git pull https://github.com/juju-solutions/bigtop mahout Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/123.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #123 commit f61ab1831caaed7f0da68a37a07076c480f411c3 Author: Konstantinos Tsakalozos <konstantinos.tsakalozos@canonical.com> Date: 2016-06-07T14:01:49Z BIGTOP-2483 : Add Mahout Charm
          Hide
          ktsakalozos Konstantinos Tsakalozos added a comment -

          BIGTOP-2473 is already merged

          Show
          ktsakalozos Konstantinos Tsakalozos added a comment - BIGTOP-2473 is already merged
          Hide
          ktsakalozos Konstantinos Tsakalozos added a comment -

          The charm has undergone two internal reviews by Andrew McLeod

          Show
          ktsakalozos Konstantinos Tsakalozos added a comment - The charm has undergone two internal reviews by Andrew McLeod
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kwmonroe commented on the issue:

          https://github.com/apache/bigtop/pull/123

          Hey Kostas, this doesn't feel right to me. I've added a hadoop-client -> openjdk relation, followed by a hadoop-client -> mahout relation, but mahout still reports "Waiting for Java". It's only after I explicitly add an openjdk -> mahout relation that the charm deploys.

          Is this expected behavior? If it is, I don't like it! The principal (hadoop-client) already has openjdk related to it, so i shouldn't have to add java -> mahout. Since you surely know that already, I'm probably doing something wrong.. Either way, let's figure it out before committing upstream.

          Show
          githubbot ASF GitHub Bot added a comment - Github user kwmonroe commented on the issue: https://github.com/apache/bigtop/pull/123 Hey Kostas, this doesn't feel right to me. I've added a hadoop-client -> openjdk relation, followed by a hadoop-client -> mahout relation, but mahout still reports "Waiting for Java". It's only after I explicitly add an openjdk -> mahout relation that the charm deploys. Is this expected behavior? If it is, I don't like it! The principal (hadoop-client) already has openjdk related to it, so i shouldn't have to add java -> mahout. Since you surely know that already, I'm probably doing something wrong.. Either way, let's figure it out before committing upstream.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user smarthi commented on a diff in the pull request:

          https://github.com/apache/bigtop/pull/123#discussion_r67579574

          — Diff: bigtop-packages/src/charm/mahout/layer-mahout/actions/smoke-test —
          @@ -0,0 +1,20 @@
          +#!/bin/bash
          +#
          +set -ex
          +
          +if hdfs dfs -stat /tmp/input/ &> /dev/null; then
          + hdfs dfs -rm -r -skipTrash /tmp/input/ || true
          +fi
          +
          +hdfs dfs -mkdir /tmp/input/
          +hdfs dfs -put resources/links-converted.txt /tmp/input/
          +hdfs dfs -put resources/users.txt /tmp/input/
          +
          +if hdfs dfs -stat temp &> /dev/null; then
          + hdfs dfs -rm -r -skipTrash temp || true
          +fi
          +if hdfs dfs -stat output &> /dev/null; then
          + hdfs dfs -rm -r -skipTrash output || true
          +fi
          +
          +hadoop jar /usr/lib/mahout/mahout-mr-0.11.1-job.jar org.apache.mahout.cf.taste.hadoop.item.RecommenderJob -Dmapred.input.dir=/tmp/input/links-converted.txt -Dmapred.output.dir=output --usersFile /tmp/input/users.txt --booleanData -s SIMILARITY_LOGLIKELIHOOD
          — End diff –

          we now have mahout 0.12.2, u may wanna go with that

          Show
          githubbot ASF GitHub Bot added a comment - Github user smarthi commented on a diff in the pull request: https://github.com/apache/bigtop/pull/123#discussion_r67579574 — Diff: bigtop-packages/src/charm/mahout/layer-mahout/actions/smoke-test — @@ -0,0 +1,20 @@ +#!/bin/bash +# +set -ex + +if hdfs dfs -stat /tmp/input/ &> /dev/null; then + hdfs dfs -rm -r -skipTrash /tmp/input/ || true +fi + +hdfs dfs -mkdir /tmp/input/ +hdfs dfs -put resources/links-converted.txt /tmp/input/ +hdfs dfs -put resources/users.txt /tmp/input/ + +if hdfs dfs -stat temp &> /dev/null; then + hdfs dfs -rm -r -skipTrash temp || true +fi +if hdfs dfs -stat output &> /dev/null; then + hdfs dfs -rm -r -skipTrash output || true +fi + +hadoop jar /usr/lib/mahout/mahout-mr-0.11.1-job.jar org.apache.mahout.cf.taste.hadoop.item.RecommenderJob -Dmapred.input.dir=/tmp/input/links-converted.txt -Dmapred.output.dir=output --usersFile /tmp/input/users.txt --booleanData -s SIMILARITY_LOGLIKELIHOOD — End diff – we now have mahout 0.12.2, u may wanna go with that
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user ktsakalozos commented on a diff in the pull request:

          https://github.com/apache/bigtop/pull/123#discussion_r67843645

          — Diff: bigtop-packages/src/charm/mahout/layer-mahout/actions/smoke-test —
          @@ -0,0 +1,20 @@
          +#!/bin/bash
          +#
          +set -ex
          +
          +if hdfs dfs -stat /tmp/input/ &> /dev/null; then
          + hdfs dfs -rm -r -skipTrash /tmp/input/ || true
          +fi
          +
          +hdfs dfs -mkdir /tmp/input/
          +hdfs dfs -put resources/links-converted.txt /tmp/input/
          +hdfs dfs -put resources/users.txt /tmp/input/
          +
          +if hdfs dfs -stat temp &> /dev/null; then
          + hdfs dfs -rm -r -skipTrash temp || true
          +fi
          +if hdfs dfs -stat output &> /dev/null; then
          + hdfs dfs -rm -r -skipTrash output || true
          +fi
          +
          +hadoop jar /usr/lib/mahout/mahout-mr-0.11.1-job.jar org.apache.mahout.cf.taste.hadoop.item.RecommenderJob -Dmapred.input.dir=/tmp/input/links-converted.txt -Dmapred.output.dir=output --usersFile /tmp/input/users.txt --booleanData -s SIMILARITY_LOGLIKELIHOOD
          — End diff –

          As far as I can tell Bigtop does not yet package Mahout 0.12.2.

          So, I updated the action script and used a wildcard instead of a fixed Mahout version. This was tested against a manual deployment of 0.12.2 Mahout. Unless there are significant changes in either the packaging of Mahout or the internals of the jar, the test should be valid for both the current and future Mahout versions.

          Thank you for spotting this and helping improve the quality of the code.

          Show
          githubbot ASF GitHub Bot added a comment - Github user ktsakalozos commented on a diff in the pull request: https://github.com/apache/bigtop/pull/123#discussion_r67843645 — Diff: bigtop-packages/src/charm/mahout/layer-mahout/actions/smoke-test — @@ -0,0 +1,20 @@ +#!/bin/bash +# +set -ex + +if hdfs dfs -stat /tmp/input/ &> /dev/null; then + hdfs dfs -rm -r -skipTrash /tmp/input/ || true +fi + +hdfs dfs -mkdir /tmp/input/ +hdfs dfs -put resources/links-converted.txt /tmp/input/ +hdfs dfs -put resources/users.txt /tmp/input/ + +if hdfs dfs -stat temp &> /dev/null; then + hdfs dfs -rm -r -skipTrash temp || true +fi +if hdfs dfs -stat output &> /dev/null; then + hdfs dfs -rm -r -skipTrash output || true +fi + +hadoop jar /usr/lib/mahout/mahout-mr-0.11.1-job.jar org.apache.mahout.cf.taste.hadoop.item.RecommenderJob -Dmapred.input.dir=/tmp/input/links-converted.txt -Dmapred.output.dir=output --usersFile /tmp/input/users.txt --booleanData -s SIMILARITY_LOGLIKELIHOOD — End diff – As far as I can tell Bigtop does not yet package Mahout 0.12.2. So, I updated the action script and used a wildcard instead of a fixed Mahout version. This was tested against a manual deployment of 0.12.2 Mahout. Unless there are significant changes in either the packaging of Mahout or the internals of the jar, the test should be valid for both the current and future Mahout versions. Thank you for spotting this and helping improve the quality of the code.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user ktsakalozos commented on the issue:

          https://github.com/apache/bigtop/pull/123

          Hi Kevin, as per our yesterday's discussion we agreed to favour explicit relations even if they seem redundant. The user is indeed required to add a relation between openjdk and mahout. If we are to remove that requirement charm authors that want their charms to relate to mahout would also need to relate to java and make sure java is available before mahout deployment is triggered.

          Show
          githubbot ASF GitHub Bot added a comment - Github user ktsakalozos commented on the issue: https://github.com/apache/bigtop/pull/123 Hi Kevin, as per our yesterday's discussion we agreed to favour explicit relations even if they seem redundant. The user is indeed required to add a relation between openjdk and mahout. If we are to remove that requirement charm authors that want their charms to relate to mahout would also need to relate to java and make sure java is available before mahout deployment is triggered.
          Hide
          ktsakalozos Konstantinos Tsakalozos added a comment -

          PR request submitted

          Show
          ktsakalozos Konstantinos Tsakalozos added a comment - PR request submitted
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user petevg commented on the issue:

          https://github.com/apache/bigtop/pull/123

          Tests complete successfully, with the latest juju 2.0 beta, against AWS. I am +1 on this.

          Show
          githubbot ASF GitHub Bot added a comment - Github user petevg commented on the issue: https://github.com/apache/bigtop/pull/123 Tests complete successfully, with the latest juju 2.0 beta, against AWS. I am +1 on this.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/bigtop/pull/123

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/bigtop/pull/123
          Hide
          kwmonroe Kevin W Monroe added a comment -

          Thanks for the charm Kostas, and for the review/verification Pete!

          I had some tweaks to the test case to explicitly set 'xenial' since that's the only series we currently have available. This charm looks good:

          https://jujucharms.com/mahout/

          Show
          kwmonroe Kevin W Monroe added a comment - Thanks for the charm Kostas, and for the review/verification Pete! I had some tweaks to the test case to explicitly set 'xenial' since that's the only series we currently have available. This charm looks good: https://jujucharms.com/mahout/

            People

            • Assignee:
              ktsakalozos Konstantinos Tsakalozos
              Reporter:
              ktsakalozos Konstantinos Tsakalozos
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development