Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.1
    • Component/s: build
    • Labels:
      None

      Description

      We released 0.11.2 of Mahout this weekend.

        Issue Links

          Activity

          Hide
          evans_ye Evans Ye added a comment -

          Thanks for the patch, Andrew Musselman.
          Closing as fixed.

          Show
          evans_ye Evans Ye added a comment - Thanks for the patch, Andrew Musselman . Closing as fixed.
          Hide
          andrew.musselman Andrew Musselman added a comment -

          I'd say so; thanks Evans.

          Show
          andrew.musselman Andrew Musselman added a comment - I'd say so; thanks Evans.
          Hide
          evans_ye Evans Ye added a comment -

          It seems out CI is doing fine with this.
          May I close this as fixed?

          Show
          evans_ye Evans Ye added a comment - It seems out CI is doing fine with this. May I close this as fixed?
          Hide
          rvs Roman Shaposhnik added a comment -

          I've just committed the patch. Lets see what our CI does.

          Show
          rvs Roman Shaposhnik added a comment - I've just committed the patch. Lets see what our CI does.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/bigtop/pull/199

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/bigtop/pull/199
          Hide
          andrew.musselman Andrew Musselman added a comment -

          Thanks, very helpful.

          Show
          andrew.musselman Andrew Musselman added a comment - Thanks, very helpful.
          Hide
          evans_ye Evans Ye added a comment -

          I've edited the doc slightly to prevent misunderstanding:
          https://cwiki.apache.org/confluence/display/BIGTOP/How+to+build+Bigtop-trunk

          To answer your question, Andrew Musselman, There're several way to evaluate the package:

          • The most solid way
            After packages built, using Bigtop Provisioner(Vagrant or Docker) to get the local built package provisioned and then run mahout iTest to make sure the functionality are all good. However, I'm not quite sure whether everything is ready on this path.
          • The naive way
            apt-get install mahout and ran some examples manually.
            I myself ofter go for the later...

          Regarding to the docker question, technically speaking, the docker build DOES touch the underlying system because we bind mount the bigtop repo into container with -v option. All the build result in bigtop/output inside container will be available outside of the container as well.
          To undo any effects? I'm not quite sure, but you can just remove bigtop/output. If you'd like to download newer version of upstream tar.gz, remove bigtop/dl as well.

          Show
          evans_ye Evans Ye added a comment - I've edited the doc slightly to prevent misunderstanding: https://cwiki.apache.org/confluence/display/BIGTOP/How+to+build+Bigtop-trunk To answer your question, Andrew Musselman , There're several way to evaluate the package: The most solid way After packages built, using Bigtop Provisioner(Vagrant or Docker) to get the local built package provisioned and then run mahout iTest to make sure the functionality are all good. However, I'm not quite sure whether everything is ready on this path. The naive way apt-get install mahout and ran some examples manually. I myself ofter go for the later... Regarding to the docker question, technically speaking, the docker build DOES touch the underlying system because we bind mount the bigtop repo into container with -v option. All the build result in bigtop/output inside container will be available outside of the container as well. To undo any effects? I'm not quite sure, but you can just remove bigtop/output. If you'd like to download newer version of upstream tar.gz, remove bigtop/dl as well.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user c0s commented on the issue:

          https://github.com/apache/bigtop/pull/199

          Seems LGTM

          Show
          githubbot ASF GitHub Bot added a comment - Github user c0s commented on the issue: https://github.com/apache/bigtop/pull/199 Seems LGTM
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user andrewmusselman opened a pull request:

          https://github.com/apache/bigtop/pull/199

          BIGTOP-2355: Update Mahout version to 0.13.0

          Tested with deb, rpm, pkg, and tar; expected files are all created but was unable to test deb install due to a hadoop-client versioning issue detailed on the dev list.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/andrewmusselman/bigtop BIGTOP-2355

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/bigtop/pull/199.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #199


          commit da1ba0acc60d6a6d9930289b3533ce6b7e8d80da
          Author: Andrew Musselman <akm@apache.org>
          Date: 2017-04-26T04:51:14Z

          BIGTOP-2355: Update Mahout version to 0.13.0


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user andrewmusselman opened a pull request: https://github.com/apache/bigtop/pull/199 BIGTOP-2355 : Update Mahout version to 0.13.0 Tested with deb, rpm, pkg, and tar; expected files are all created but was unable to test deb install due to a hadoop-client versioning issue detailed on the dev list. You can merge this pull request into a Git repository by running: $ git pull https://github.com/andrewmusselman/bigtop BIGTOP-2355 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/199.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #199 commit da1ba0acc60d6a6d9930289b3533ce6b7e8d80da Author: Andrew Musselman <akm@apache.org> Date: 2017-04-26T04:51:14Z BIGTOP-2355 : Update Mahout version to 0.13.0
          Hide
          andrew.musselman Andrew Musselman added a comment -

          In a bigger sense, does this docker build touch the underlying system, or is it self-contained so to speak? And is there any standard way of undoing any effects it has?

          Thanks

          Show
          andrew.musselman Andrew Musselman added a comment - In a bigger sense, does this docker build touch the underlying system, or is it self-contained so to speak? And is there any standard way of undoing any effects it has? Thanks
          Hide
          andrew.musselman Andrew Musselman added a comment -

          Ah yes, it's coming back to me now.

          It's from here and I just replaced a token with 'mahout' hoping it'd work: https://cwiki.apache.org/confluence/display/BIGTOP/How+to+build+Bigtop-trunk

          However assuming it builds, where/how would I run the newly constructed mahout package? That would be a good comment at the end of that wiki page I think.

          Thanks!

          Show
          andrew.musselman Andrew Musselman added a comment - Ah yes, it's coming back to me now. It's from here and I just replaced a token with 'mahout' hoping it'd work: https://cwiki.apache.org/confluence/display/BIGTOP/How+to+build+Bigtop-trunk However assuming it builds, where/how would I run the newly constructed mahout package? That would be a good comment at the end of that wiki page I think. Thanks!
          Hide
          evans_ye Evans Ye added a comment - - edited

          Andrew Musselman your command looks a bit wrong:

          $ docker run -u jenkins --rm --workdir /ws -v `pwd`:/ws bigtop/slaves:trunk-ubuntu-14.04 bash -l -c './gradlew bigtop-mahout-pkg'
          

          The gradle task should be:

          $ docker run -u jenkins --rm --workdir /ws -v `pwd`:/ws bigtop/slaves:trunk-ubuntu-14.04 bash -l -c './gradlew mahout-pkg'
          

          You can call ./gradlew tasks to find out the correct task to invoke.

          Once you built successfully, you can find packages:

          $ ls -l output/mahout/
          total 227892
          -rw-r--r-- 1 jenkins jenkins 222311754 Apr 23 08:59 mahout_0.12.2-1_all.deb
          -rw-r--r-- 1 jenkins jenkins   5669356 Apr 23 09:00 mahout_0.12.2-1_amd64.build
          -rw-r--r-- 1 jenkins jenkins      1001 Apr 23 08:59 mahout_0.12.2-1_amd64.changes
          -rw-r--r-- 1 jenkins jenkins      6839 Apr 23 08:51 mahout_0.12.2-1.debian.tar.gz
          -rw-r--r-- 1 jenkins jenkins       840 Apr 23 08:51 mahout_0.12.2-1.dsc
          -rw-r--r-- 1 jenkins jenkins      1230 Apr 23 08:51 mahout_0.12.2-1_source.changes
          -rw-r--r-- 1 jenkins jenkins   4821539 Apr 23 08:51 mahout_0.12.2.orig.tar.gz
          -rw-r--r-- 1 jenkins jenkins    522102 Apr 23 08:59 mahout-doc_0.12.2-1_all.deb
          

          Meanwhile, I didn't tested, but you probably need to make sure that your host machine do have jenkins with uid=1000.

          BTW, where did you get that docker command from?
          I'd like to polish it so that there won't be confuse again.

          Show
          evans_ye Evans Ye added a comment - - edited Andrew Musselman your command looks a bit wrong: $ docker run -u jenkins --rm --workdir /ws -v `pwd`:/ws bigtop/slaves:trunk-ubuntu-14.04 bash -l -c './gradlew bigtop-mahout-pkg' The gradle task should be: $ docker run -u jenkins --rm --workdir /ws -v `pwd`:/ws bigtop/slaves:trunk-ubuntu-14.04 bash -l -c './gradlew mahout-pkg' You can call ./gradlew tasks to find out the correct task to invoke. Once you built successfully, you can find packages: $ ls -l output/mahout/ total 227892 -rw-r--r-- 1 jenkins jenkins 222311754 Apr 23 08:59 mahout_0.12.2-1_all.deb -rw-r--r-- 1 jenkins jenkins 5669356 Apr 23 09:00 mahout_0.12.2-1_amd64.build -rw-r--r-- 1 jenkins jenkins 1001 Apr 23 08:59 mahout_0.12.2-1_amd64.changes -rw-r--r-- 1 jenkins jenkins 6839 Apr 23 08:51 mahout_0.12.2-1.debian.tar.gz -rw-r--r-- 1 jenkins jenkins 840 Apr 23 08:51 mahout_0.12.2-1.dsc -rw-r--r-- 1 jenkins jenkins 1230 Apr 23 08:51 mahout_0.12.2-1_source.changes -rw-r--r-- 1 jenkins jenkins 4821539 Apr 23 08:51 mahout_0.12.2.orig.tar.gz -rw-r--r-- 1 jenkins jenkins 522102 Apr 23 08:59 mahout-doc_0.12.2-1_all.deb Meanwhile, I didn't tested, but you probably need to make sure that your host machine do have jenkins with uid=1000. BTW, where did you get that docker command from? I'd like to polish it so that there won't be confuse again.
          Hide
          evans_ye Evans Ye added a comment -

          AFAIK we put all the component specific build steps in do-component-build, so every component's do-component-build looks largely different from each other. There's no formalized build framework to pass something from user down to mvn. Hence I guess the answer is no.

          Show
          evans_ye Evans Ye added a comment - AFAIK we put all the component specific build steps in do-component-build, so every component's do-component-build looks largely different from each other. There's no formalized build framework to pass something from user down to mvn. Hence I guess the answer is no.
          Hide
          andrew.musselman Andrew Musselman added a comment -

          Also, is there a way for users to pass in maven profiles (-Pfoo) when building?

          I see this is where mahout is built:
          $ grep -r mahout | grep mvn
          bigtop-packages/src/common/mahout/do-component-build:mvn clean install -DskipTests -Dmahout.skip.distribution=false \

          Show
          andrew.musselman Andrew Musselman added a comment - Also, is there a way for users to pass in maven profiles (-Pfoo) when building? I see this is where mahout is built: $ grep -r mahout | grep mvn bigtop-packages/src/common/mahout/do-component-build:mvn clean install -DskipTests -Dmahout.skip.distribution=false \
          Hide
          andrew.musselman Andrew Musselman added a comment -

          Roman Shaposhnik The edits are minor but I got lost in how to test builds; I see on here there's a docker method: https://cwiki.apache.org/confluence/display/BIGTOP/How+to+build+Bigtop-trunk

          And I'm running: $ docker run -u jenkins --rm --workdir /ws -v `pwd`:/ws bigtop/slaves:trunk-ubuntu-14.04 bash -l -c './gradlew bigtop-mahout-pkg'

          Where does the package get built and what's the path to run a built package?

          Thanks!

          Show
          andrew.musselman Andrew Musselman added a comment - Roman Shaposhnik The edits are minor but I got lost in how to test builds; I see on here there's a docker method: https://cwiki.apache.org/confluence/display/BIGTOP/How+to+build+Bigtop-trunk And I'm running: $ docker run -u jenkins --rm --workdir /ws -v `pwd`:/ws bigtop/slaves:trunk-ubuntu-14.04 bash -l -c './gradlew bigtop-mahout-pkg' Where does the package get built and what's the path to run a built package? Thanks!
          Hide
          andrew.musselman Andrew Musselman added a comment -

          Go ahead without us; we found a build issue that is going to take some time; hopefully we'll be done by the time you do a point release.

          Thanks!

          Show
          andrew.musselman Andrew Musselman added a comment - Go ahead without us; we found a build issue that is going to take some time; hopefully we'll be done by the time you do a point release. Thanks!
          Hide
          rvs Roman Shaposhnik added a comment -

          Looking at how vote is progressing I'm afraid this one needs to be pushed to 1.3.0. Alternatively we can do 1.2.1 with a few components that wanted to make it to 1.2.0 but were late.

          Show
          rvs Roman Shaposhnik added a comment - Looking at how vote is progressing I'm afraid this one needs to be pushed to 1.3.0. Alternatively we can do 1.2.1 with a few components that wanted to make it to 1.2.0 but were late.
          Hide
          andrew.musselman Andrew Musselman added a comment -

          Roman Shaposhnik Gotcha; let's see how fast we can vote, this was a minor fix.

          Show
          andrew.musselman Andrew Musselman added a comment - Roman Shaposhnik Gotcha; let's see how fast we can vote, this was a minor fix.
          Hide
          rvs Roman Shaposhnik added a comment -

          Andrew Musselman in that case, I think we'd have to move ahead with the first RC of Bigtop as-is. I'll start an email on dev@ asking for how late can we allow leaf node components like Mahout to be bumped.

          Show
          rvs Roman Shaposhnik added a comment - Andrew Musselman in that case, I think we'd have to move ahead with the first RC of Bigtop as-is. I'll start an email on dev@ asking for how late can we allow leaf node components like Mahout to be bumped.
          Hide
          andrew.musselman Andrew Musselman added a comment -

          Probably Wednesday/Thursday

          Show
          andrew.musselman Andrew Musselman added a comment - Probably Wednesday/Thursday
          Hide
          rvs Roman Shaposhnik added a comment -

          Andrew Musselman it seems you guys are re-spinning. What's the new ETA?

          Show
          rvs Roman Shaposhnik added a comment - Andrew Musselman it seems you guys are re-spinning. What's the new ETA?
          Hide
          andrew.musselman Andrew Musselman added a comment -

          We're still conducting a vote on our RC for 0.13.0, closing on Friday. If it goes through I can put a patch in over the weekend.

          Show
          andrew.musselman Andrew Musselman added a comment - We're still conducting a vote on our RC for 0.13.0, closing on Friday. If it goes through I can put a patch in over the weekend.
          Hide
          rvs Roman Shaposhnik added a comment -

          Andrew Musselman ping! any chance we can see a patch in the next couple of days?

          Show
          rvs Roman Shaposhnik added a comment - Andrew Musselman ping! any chance we can see a patch in the next couple of days?
          Hide
          cos Konstantin Boudnik added a comment -

          Well, we have about 40 issues open for the release. A couple of weeks, I'd say.

          Show
          cos Konstantin Boudnik added a comment - Well, we have about 40 issues open for the release. A couple of weeks, I'd say.
          Hide
          andrew.musselman Andrew Musselman added a comment -

          Yep, what's your timeframe for the 1.2 release?

          Show
          andrew.musselman Andrew Musselman added a comment - Yep, what's your timeframe for the 1.2 release?
          Hide
          cos Konstantin Boudnik added a comment -

          Andrew Musselman, could you please make sure the upgrade (to whatever is the latest going out) is done on time for 1.2? Thanks!

          Show
          cos Konstantin Boudnik added a comment - Andrew Musselman , could you please make sure the upgrade (to whatever is the latest going out) is done on time for 1.2? Thanks!
          Hide
          cos Konstantin Boudnik added a comment -

          I can not recall any immediate plans to cut the release. Perhaps in October/November so we can update the stack before the year end. Thanks!

          Show
          cos Konstantin Boudnik added a comment - I can not recall any immediate plans to cut the release. Perhaps in October/November so we can update the stack before the year end. Thanks!
          Hide
          andrew.musselman Andrew Musselman added a comment -

          0.13 is coming out soon, we'll rev this then unless you have a release imminent.

          Show
          andrew.musselman Andrew Musselman added a comment - 0.13 is coming out soon, we'll rev this then unless you have a release imminent.
          Hide
          andrew.musselman Andrew Musselman added a comment -

          0.12.2 is latest

          Show
          andrew.musselman Andrew Musselman added a comment - 0.12.2 is latest
          Hide
          andrew.musselman Andrew Musselman added a comment -

          Will bump to 0.12.1 after our release this weekend.

          Show
          andrew.musselman Andrew Musselman added a comment - Will bump to 0.12.1 after our release this weekend.
          Hide
          andrew.musselman Andrew Musselman added a comment -

          Agreed

          Show
          andrew.musselman Andrew Musselman added a comment - Agreed
          Hide
          smarthi Suneel Marthi added a comment - - edited

          I would hold off from working on this in light of Mahout 0.12.0 release that should be out in next 2 weeks that adds Apache Flink as a backend execution engine.

          Mahout 0.12.0 would be dependent on merging PR - https://github.com/apache/bigtop/pull/93 into BigTop

          Show
          smarthi Suneel Marthi added a comment - - edited I would hold off from working on this in light of Mahout 0.12.0 release that should be out in next 2 weeks that adds Apache Flink as a backend execution engine. Mahout 0.12.0 would be dependent on merging PR - https://github.com/apache/bigtop/pull/93 into BigTop

            People

            • Assignee:
              andrew.musselman Andrew Musselman
              Reporter:
              andrew.musselman Andrew Musselman
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development