Uploaded image for project: 'Mahout'
  1. Mahout
  2. MAHOUT-1706

Remove dependency jars from /lib in mahout binary distribution

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 0.10.0
    • Fix Version/s: 1.0.0
    • Component/s: None
    • Labels:
      None

      Description

      The mahout distribution currently is shipping ~56 MB of dependecy jars in the /lib directory of the distribution. These are only added to the classpath by /bin/mahout in the binary distribution. It seems that we can remove them from the distribution. (we need to get the size of the distribution down)

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user andrewpalumbo closed the pull request at:

          https://github.com/apache/mahout/pull/129

          Show
          githubbot ASF GitHub Bot added a comment - Github user andrewpalumbo closed the pull request at: https://github.com/apache/mahout/pull/129
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user andrewpalumbo reopened a pull request:

          https://github.com/apache/mahout/pull/129

          MAHOUT-1706: remove dependency jars from /lib in the binary distribution

          The mahout distribution currently is shipping ~56 MB of dependecy jars in the /lib directory of the distribution. These are only added to the classpath by /bin/mahout in the binary distribution. It seems that we can remove them from the distribution. (we need to get the size of the distribution down)

          Any input is appreciated.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/andrewpalumbo/mahout ex-dep3

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/mahout/pull/129.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #129


          commit 7064fdfdc3bad241f6fc6092edd6cd4a55b532f0
          Author: Andrew Palumbo <apalumbo@apache.org>
          Date: 2015-05-06T00:01:03Z

          remove dependency jars from /lib in distribution


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user andrewpalumbo reopened a pull request: https://github.com/apache/mahout/pull/129 MAHOUT-1706 : remove dependency jars from /lib in the binary distribution The mahout distribution currently is shipping ~56 MB of dependecy jars in the /lib directory of the distribution. These are only added to the classpath by /bin/mahout in the binary distribution. It seems that we can remove them from the distribution. (we need to get the size of the distribution down) Any input is appreciated. You can merge this pull request into a Git repository by running: $ git pull https://github.com/andrewpalumbo/mahout ex-dep3 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/mahout/pull/129.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #129 commit 7064fdfdc3bad241f6fc6092edd6cd4a55b532f0 Author: Andrew Palumbo <apalumbo@apache.org> Date: 2015-05-06T00:01:03Z remove dependency jars from /lib in distribution
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user andrewpalumbo commented on the pull request:

          https://github.com/apache/mahout/pull/129#issuecomment-106882081

          Closing this PR (against master). Will re-open it against the 0.10.x branch. This still needs testing on a cluster.

          Show
          githubbot ASF GitHub Bot added a comment - Github user andrewpalumbo commented on the pull request: https://github.com/apache/mahout/pull/129#issuecomment-106882081 Closing this PR (against master). Will re-open it against the 0.10.x branch. This still needs testing on a cluster.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user andrewpalumbo closed the pull request at:

          https://github.com/apache/mahout/pull/129

          Show
          githubbot ASF GitHub Bot added a comment - Github user andrewpalumbo closed the pull request at: https://github.com/apache/mahout/pull/129
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user andrewpalumbo opened a pull request:

          https://github.com/apache/mahout/pull/133

          MAHOUT-1706-0.10.x

          The mahout distribution currently is shipping ~56 MB of dependency jars in the /lib directory of the distribution. These are only added to the classpath by /bin/mahout in the binary distribution. It seems that we can remove them from the distribution. (we need to get the size of the distribution down).

          This branch still needs testing on an cluster.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/andrewpalumbo/mahout MAHOUT-1706-0.10.x

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/mahout/pull/133.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #133


          commit 6d4f76fe2785e0998426fbb7b3f6d5a38dca971e
          Author: Andrew Palumbo <apalumbo@apache.org>
          Date: 2015-05-06T00:01:03Z

          remove dependency jars from /lib in distribution

          commit b1dca2e37145f39ee15a0ad085f569a388114772
          Author: Andrew Palumbo <apalumbo@apache.org>
          Date: 2015-05-07T19:50:49Z

          remove example/target/dependency/*.jar from the /bin/mahout classpath

          commit a65e6d7336130ede8dfd2d61ec2a4f3261069bfb
          Author: Andrew Palumbo <apalumbo@apache.org>
          Date: 2015-05-08T02:09:37Z

          pickup hadoop jars in development mode


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user andrewpalumbo opened a pull request: https://github.com/apache/mahout/pull/133 MAHOUT-1706 -0.10.x The mahout distribution currently is shipping ~56 MB of dependency jars in the /lib directory of the distribution. These are only added to the classpath by /bin/mahout in the binary distribution. It seems that we can remove them from the distribution. (we need to get the size of the distribution down). This branch still needs testing on an cluster. You can merge this pull request into a Git repository by running: $ git pull https://github.com/andrewpalumbo/mahout MAHOUT-1706 -0.10.x Alternatively you can review and apply these changes as the patch at: https://github.com/apache/mahout/pull/133.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #133 commit 6d4f76fe2785e0998426fbb7b3f6d5a38dca971e Author: Andrew Palumbo <apalumbo@apache.org> Date: 2015-05-06T00:01:03Z remove dependency jars from /lib in distribution commit b1dca2e37145f39ee15a0ad085f569a388114772 Author: Andrew Palumbo <apalumbo@apache.org> Date: 2015-05-07T19:50:49Z remove example/target/dependency/*.jar from the /bin/mahout classpath commit a65e6d7336130ede8dfd2d61ec2a4f3261069bfb Author: Andrew Palumbo <apalumbo@apache.org> Date: 2015-05-08T02:09:37Z pickup hadoop jars in development mode
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user dlyubimov commented on the pull request:

          https://github.com/apache/mahout/pull/133#issuecomment-110121739

          noted...
          this does or does not affect shell classpath?

          Show
          githubbot ASF GitHub Bot added a comment - Github user dlyubimov commented on the pull request: https://github.com/apache/mahout/pull/133#issuecomment-110121739 noted... this does or does not affect shell classpath?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user andrewpalumbo commented on the pull request:

          https://github.com/apache/mahout/pull/133#issuecomment-110522411

          It should not (negatively) affect the shell classpath, since all of the necessary dependencies are either packaged with mahout-spark-*.jars or in the classpath returned by {{$

          {SPARK_HOME}

          /bin/compute-classpath.sh}}.

          This patch actually does not affect the classpath of a source build at all. It only prevents the release process from packaging dependency jars into the /lib dir of the binary distribution artifact.

          However since it does add all of the jars in the /lib directory of the binary build for any mahout command line flavor (spark, MR, h2o), I have been hesitant to push it without more extensive testing (cluster and local).

          I am going to close this for now and investigate a better approach.

          Show
          githubbot ASF GitHub Bot added a comment - Github user andrewpalumbo commented on the pull request: https://github.com/apache/mahout/pull/133#issuecomment-110522411 It should not (negatively) affect the shell classpath, since all of the necessary dependencies are either packaged with mahout-spark-*.jars or in the classpath returned by {{$ {SPARK_HOME} /bin/compute-classpath.sh}}. This patch actually does not affect the classpath of a source build at all. It only prevents the release process from packaging dependency jars into the /lib dir of the binary distribution artifact. However since it does add all of the jars in the /lib directory of the binary build for any mahout command line flavor (spark, MR, h2o), I have been hesitant to push it without more extensive testing (cluster and local). I am going to close this for now and investigate a better approach.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user andrewpalumbo commented on the pull request:

          https://github.com/apache/mahout/pull/133#issuecomment-110522528

          closing without pushing while looking at different approaches.

          Show
          githubbot ASF GitHub Bot added a comment - Github user andrewpalumbo commented on the pull request: https://github.com/apache/mahout/pull/133#issuecomment-110522528 closing without pushing while looking at different approaches.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user andrewpalumbo closed the pull request at:

          https://github.com/apache/mahout/pull/133

          Show
          githubbot ASF GitHub Bot added a comment - Github user andrewpalumbo closed the pull request at: https://github.com/apache/mahout/pull/133
          Hide
          andrew.musselman Andrew Musselman added a comment -

          Is this finished?

          Show
          andrew.musselman Andrew Musselman added a comment - Is this finished?
          Hide
          Andrew_Palumbo Andrew Palumbo added a comment -

          Not finished, but is not as high of a priority as when reported.. We needed at the time to get the build < 200M.. that is done. This should be addressed at some point though. Relates to MAHOUT-1705.

          Show
          Andrew_Palumbo Andrew Palumbo added a comment - Not finished, but is not as high of a priority as when reported.. We needed at the time to get the build < 200M.. that is done. This should be addressed at some point though. Relates to MAHOUT-1705 .
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user andrewpalumbo reopened a pull request:

          https://github.com/apache/mahout/pull/129

          MAHOUT-1706: remove dependency jars from /lib in the binary distribution

          The mahout distribution currently is shipping ~56 MB of dependecy jars in the /lib directory of the distribution. These are only added to the classpath by /bin/mahout in the binary distribution. It seems that we can remove them from the distribution. (we need to get the size of the distribution down)

          Any input is appreciated.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/andrewpalumbo/mahout ex-dep3

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/mahout/pull/129.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #129


          commit 7064fdfdc3bad241f6fc6092edd6cd4a55b532f0
          Author: Andrew Palumbo <apalumbo@apache.org>
          Date: 2015-05-06T00:01:03Z

          remove dependency jars from /lib in distribution

          commit abdb4cf0896289cdb2083ef19def006cb2bf9f9d
          Author: Andrew Palumbo <apalumbo@apache.org>
          Date: 2015-05-07T19:50:49Z

          remove example/target/dependency/*.jar from the /bin/mahout classpath

          commit 239bcaf384385b454fad5dd4962a622e03855438
          Author: Andrew Palumbo <apalumbo@apache.org>
          Date: 2015-05-07T20:01:28Z

          Merge branch 'master' into ex-dep3

          commit ff13df2d7e67658a9da4b583584cfd9b3d202a15
          Author: Andrew Palumbo <apalumbo@apache.org>
          Date: 2015-05-08T02:09:37Z

          pickup hadoop jars in development mode


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user andrewpalumbo reopened a pull request: https://github.com/apache/mahout/pull/129 MAHOUT-1706 : remove dependency jars from /lib in the binary distribution The mahout distribution currently is shipping ~56 MB of dependecy jars in the /lib directory of the distribution. These are only added to the classpath by /bin/mahout in the binary distribution. It seems that we can remove them from the distribution. (we need to get the size of the distribution down) Any input is appreciated. You can merge this pull request into a Git repository by running: $ git pull https://github.com/andrewpalumbo/mahout ex-dep3 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/mahout/pull/129.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #129 commit 7064fdfdc3bad241f6fc6092edd6cd4a55b532f0 Author: Andrew Palumbo <apalumbo@apache.org> Date: 2015-05-06T00:01:03Z remove dependency jars from /lib in distribution commit abdb4cf0896289cdb2083ef19def006cb2bf9f9d Author: Andrew Palumbo <apalumbo@apache.org> Date: 2015-05-07T19:50:49Z remove example/target/dependency/*.jar from the /bin/mahout classpath commit 239bcaf384385b454fad5dd4962a622e03855438 Author: Andrew Palumbo <apalumbo@apache.org> Date: 2015-05-07T20:01:28Z Merge branch 'master' into ex-dep3 commit ff13df2d7e67658a9da4b583584cfd9b3d202a15 Author: Andrew Palumbo <apalumbo@apache.org> Date: 2015-05-08T02:09:37Z pickup hadoop jars in development mode
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user smarthi commented on the issue:

          https://github.com/apache/mahout/pull/129

          Try running this thru

          `mvn -Pmahout-release,apache-release,hadoop2 package`

          to ensure that the build goes thru fine and verify the size of the artifacts in /target after this.

          Show
          githubbot ASF GitHub Bot added a comment - Github user smarthi commented on the issue: https://github.com/apache/mahout/pull/129 Try running this thru `mvn -Pmahout-release,apache-release,hadoop2 package` to ensure that the build goes thru fine and verify the size of the artifacts in /target after this.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user smarthi commented on the issue:

          https://github.com/apache/mahout/pull/129

          Is this good to merge ?

          Show
          githubbot ASF GitHub Bot added a comment - Github user smarthi commented on the issue: https://github.com/apache/mahout/pull/129 Is this good to merge ?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user andrewmusselman commented on the issue:

          https://github.com/apache/mahout/pull/129

          Where is this, merge or no?

          Show
          githubbot ASF GitHub Bot added a comment - Github user andrewmusselman commented on the issue: https://github.com/apache/mahout/pull/129 Where is this, merge or no?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user andrewpalumbo commented on the issue:

          https://github.com/apache/mahout/pull/129

          I just reopened it this PR after discussion last week. Haven't had a chance to test it yet.

          Show
          githubbot ASF GitHub Bot added a comment - Github user andrewpalumbo commented on the issue: https://github.com/apache/mahout/pull/129 I just reopened it this PR after discussion last week. Haven't had a chance to test it yet.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user andrewpalumbo commented on the issue:

          https://github.com/apache/mahout/pull/129

          The source builds and tests fine for me locally (Travis can't handle the mr tests). This really needs to be tested on a (pseudo)-cluster after being built with:
          `mvn -Pmahout-release,apache-release,hadoop2 package`
          for both binary and source releases.

          Show
          githubbot ASF GitHub Bot added a comment - Github user andrewpalumbo commented on the issue: https://github.com/apache/mahout/pull/129 The source builds and tests fine for me locally (Travis can't handle the mr tests). This really needs to be tested on a (pseudo)-cluster after being built with: `mvn -Pmahout-release,apache-release,hadoop2 package` for both binary and source releases.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user andrewmusselman commented on the issue:

          https://github.com/apache/mahout/pull/129

          I'll take it for a spin.

          Show
          githubbot ASF GitHub Bot added a comment - Github user andrewmusselman commented on the issue: https://github.com/apache/mahout/pull/129 I'll take it for a spin.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user rawkintrevo commented on the issue:

          https://github.com/apache/mahout/pull/129

          Close or merge (or stand by?)

          Show
          githubbot ASF GitHub Bot added a comment - Github user rawkintrevo commented on the issue: https://github.com/apache/mahout/pull/129 Close or merge (or stand by?)
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user andrewpalumbo commented on the issue:

          https://github.com/apache/mahout/pull/129

          Let's hold off on this until 0.13.1.. it will mean much more testing.. only wins us a bit of space.

          Sent from my Verizon Wireless 4G LTE smartphone

          -------- Original message --------
          From: Trevor Grant <notifications@github.com>
          Date: 02/24/2017 5:58 AM (GMT-08:00)
          To: apache/mahout <mahout@noreply.github.com>
          Cc: Andrew Palumbo <ap.dev@outlook.com>, State change <state_change@noreply.github.com>
          Subject: Re: [apache/mahout] MAHOUT-1706: remove dependency jars from /lib in the binary distribution (#129)

          Close or merge (or stand by?)

          -
          You are receiving this because you modified the open/close state.
          Reply to this email directly, view it on GitHub<https://github.com/apache/mahout/pull/129#issuecomment-282297167>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AHU2HaOERl4_73-PiILxCvvaQqWMw7rJks5rfuH-gaJpZM4EQy_I>.

          Show
          githubbot ASF GitHub Bot added a comment - Github user andrewpalumbo commented on the issue: https://github.com/apache/mahout/pull/129 Let's hold off on this until 0.13.1.. it will mean much more testing.. only wins us a bit of space. Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Trevor Grant <notifications@github.com> Date: 02/24/2017 5:58 AM (GMT-08:00) To: apache/mahout <mahout@noreply.github.com> Cc: Andrew Palumbo <ap.dev@outlook.com>, State change <state_change@noreply.github.com> Subject: Re: [apache/mahout] MAHOUT-1706 : remove dependency jars from /lib in the binary distribution (#129) Close or merge (or stand by?) - You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub< https://github.com/apache/mahout/pull/129#issuecomment-282297167 >, or mute the thread< https://github.com/notifications/unsubscribe-auth/AHU2HaOERl4_73-PiILxCvvaQqWMw7rJks5rfuH-gaJpZM4EQy_I >.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user rawkintrevo commented on the issue:

          https://github.com/apache/mahout/pull/129

          Let's push forward on this quickly if possible.

          Show
          githubbot ASF GitHub Bot added a comment - Github user rawkintrevo commented on the issue: https://github.com/apache/mahout/pull/129 Let's push forward on this quickly if possible.

            People

            • Assignee:
              Andrew_Palumbo Andrew Palumbo
              Reporter:
              Andrew_Palumbo Andrew Palumbo
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:

                Development