Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.7.0
    • Fix Version/s: 0.7.0
    • Component/s: new service
    • Labels:
      None

      Description

      Here is an initial patch to support Mahout as a Whirr service.

      I created the role 'mahout-home' which can be used to install the binary Mahout distribution on a Hadoop namenode.
      By combining this role with configuration for a Hadoop cluster you can SSH into the namenode, su to root and start running Mahout jobs via the mahout script immediately.

      The 'mahout-home' role has two properties

      Mahout version whirr.mahout.version
      URL of the Mahout binary distribution tarball whirr.mahout.tarball.url

      Note that I used a snapshot version of Mahout for testing, revision 1169784, because there were some problems with the Mahout script in 0.5 that have been fixed on trunk, see MAHOUT-680. To test you can set the tarball property to this link http://dl.dropbox.com/u/13436484/mahout-distribution-0.6-SNAPSHOT.tar.gz

      I used configure actions and the onBeforeConfigure(). If there is a better way to express this with the Whirr API let me know.

      Currently I am investigating a 'mahout-jar' role, which installs the Mahout examples job jar under $HADOOP_HOME/lib on a tasktracer node. I already have some code for putting the jar in place but when running a job from my local machine I still get ClassNotFoundExceptions. I believe this is because Hadoop has already started before the jar is put in the lib dir, so the jar won't be picked up, but I have to investigate some more. From WHIRR-221 I understood that there is no support (yet?) for ordering of services but if you have an idea on how to fix this let me know.

      Comments and suggestions welcome!

      1. WHIRR-384.patch
        27 kB
        Andrei Savu
      2. WHIRR-384-mahout-client.patch
        12 kB
        Frank Scholten
      3. WHIRR-384-mahout-home.patch
        13 kB
        Frank Scholten

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Frank Scholten
            Reporter:
            Frank Scholten
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development