Uploaded image for project: 'Mahout'
  1. Mahout
  2. MAHOUT-1939

fastutil version clash with spark distributions

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None

      Description

      Version difference in fast util breaks sparse algebra (specifically, RandomAccessSparseVector in assign, e.g., vec *= 5).

      observed version in CDH:

      file:/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.21/jars/fastutil-6.3.jar

      mahout uses 7.0.12

      java.lang.UnsupportedOperationException
      at it.unimi.dsi.fastutil.ints.AbstractInt2DoubleMap$BasicEntry.setValue(AbstractInt2DoubleMap.java:146)
      at org.apache.mahout.math.RandomAccessSparseVector$RandomAccessElement.set(RandomAccessSparseVector.java:235)
      at org.apache.mahout.math.VectorView$DecoratorElement.set(VectorView.java:181)
      at org.apache.mahout.math.AbstractVector.assign(AbstractVector.java:536)
      at org.apache.mahout.math.scalabindings.RLikeVectorOps.$div$eq(RLikeVectorOps.scala:45)
      ...

        Issue Links

          Activity

          Hide
          dlyubimov Dmitriy Lyubimov added a comment - - edited

          perhaps mahout should include fast-util in a shaded form in mahout-math or mahout-math-scala.
          like this (in mahout-math):

          <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-shade-plugin</artifactId>
          <version>3.0.0</version>
          <executions>
          <execution>
          <phase>package</phase>
          <goals>
          <goal>shade</goal>
          </goals>
          <configuration>
          <artifactSet>
          <includes>
          <include>it.unimi.dsi:fastutil</include>
          </includes>
          </artifactSet>
          <relocations>
          <relocation>
          <pattern>it.unimi.dsi.fastutil</pattern>
          <shadedPattern>shaded.it.unimi.dsi.fastutil</shadedPattern>
          </relocation>
          </relocations>
          </configuration>
          </execution>
          </executions>
          </plugin>

          Show
          dlyubimov Dmitriy Lyubimov added a comment - - edited perhaps mahout should include fast-util in a shaded form in mahout-math or mahout-math-scala. like this (in mahout-math): <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>3.0.0</version> <executions> <execution> <phase>package</phase> <goals> <goal>shade</goal> </goals> <configuration> <artifactSet> <includes> <include>it.unimi.dsi:fastutil</include> </includes> </artifactSet> <relocations> <relocation> <pattern>it.unimi.dsi.fastutil</pattern> <shadedPattern>shaded.it.unimi.dsi.fastutil</shadedPattern> </relocation> </relocations> </configuration> </execution> </executions> </plugin>
          Hide
          Andrew_Palumbo Andrew Palumbo added a comment -

          Dmitriy Lyubimov How many CDH Spark versions does this break? Is this a blocker for the upcoming release? Seems so.

          Show
          Andrew_Palumbo Andrew Palumbo added a comment - Dmitriy Lyubimov How many CDH Spark versions does this break? Is this a blocker for the upcoming release? Seems so.
          Hide
          Andrew_Palumbo Andrew Palumbo added a comment -

          Dmitriy Lyubimov I think i mad this a blocker do you agree? I will convert it to critical now, and back to a blocker if you believe that it is.

          Show
          Andrew_Palumbo Andrew Palumbo added a comment - Dmitriy Lyubimov I think i mad this a blocker do you agree? I will convert it to critical now, and back to a blocker if you believe that it is.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user andrewpalumbo opened a pull request:

          https://github.com/apache/mahout/pull/285

          MAHOUT-1939: Shade fastutil jar conflicictng with CDH Spark

          Reports of fastutils versions conflicting on CDH. Shade our fastutils in `mahout-math`.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/andrewpalumbo/mahout MAHOUT-1939

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/mahout/pull/285.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #285


          commit 4095c32e99b69f48ff5e091d612eb65569c25f73
          Author: Andrew Palumbo <apalumbo@apache.org>
          Date: 2017-02-26T07:25:35Z

          MAHOUT-1939: Shade fastutil jar conflicictng with CDH


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user andrewpalumbo opened a pull request: https://github.com/apache/mahout/pull/285 MAHOUT-1939 : Shade fastutil jar conflicictng with CDH Spark Reports of fastutils versions conflicting on CDH. Shade our fastutils in `mahout-math`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/andrewpalumbo/mahout MAHOUT-1939 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/mahout/pull/285.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #285 commit 4095c32e99b69f48ff5e091d612eb65569c25f73 Author: Andrew Palumbo <apalumbo@apache.org> Date: 2017-02-26T07:25:35Z MAHOUT-1939 : Shade fastutil jar conflicictng with CDH
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user andrewpalumbo commented on the issue:

          https://github.com/apache/mahout/pull/285

          local tests pass on this. @dlyubimov could you please test this against the CDH version you found a conflict with?

          Show
          githubbot ASF GitHub Bot added a comment - Github user andrewpalumbo commented on the issue: https://github.com/apache/mahout/pull/285 local tests pass on this. @dlyubimov could you please test this against the CDH version you found a conflict with?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user andrewpalumbo commented on the issue:

          https://github.com/apache/mahout/pull/285

          committed in 4d1464ef4d7a1d278c828d29a906e8f431a2fcb5

          Show
          githubbot ASF GitHub Bot added a comment - Github user andrewpalumbo commented on the issue: https://github.com/apache/mahout/pull/285 committed in 4d1464ef4d7a1d278c828d29a906e8f431a2fcb5
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user andrewpalumbo closed the pull request at:

          https://github.com/apache/mahout/pull/285

          Show
          githubbot ASF GitHub Bot added a comment - Github user andrewpalumbo closed the pull request at: https://github.com/apache/mahout/pull/285
          Hide
          Andrew_Palumbo Andrew Palumbo added a comment -

          committed in 4d1464ef4d7a1d278c828d29a906e8f431a2fcb5

          Show
          Andrew_Palumbo Andrew Palumbo added a comment - committed in 4d1464ef4d7a1d278c828d29a906e8f431a2fcb5
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Jenkins build Mahout-Quality #3428 (See https://builds.apache.org/job/Mahout-Quality/3428/)
          MAHOUT-1939: Shade fastutil jar conflicictng with CDH Spark #285 (apalumbo: rev 4d1464ef4d7a1d278c828d29a906e8f431a2fcb5)

          • (edit) math/pom.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Jenkins build Mahout-Quality #3428 (See https://builds.apache.org/job/Mahout-Quality/3428/ ) MAHOUT-1939 : Shade fastutil jar conflicictng with CDH Spark #285 (apalumbo: rev 4d1464ef4d7a1d278c828d29a906e8f431a2fcb5) (edit) math/pom.xml
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user andrewpalumbo commented on the issue:

          https://github.com/apache/mahout/pull/285

          +1 shipit ! thx!

          Show
          githubbot ASF GitHub Bot added a comment - Github user andrewpalumbo commented on the issue: https://github.com/apache/mahout/pull/285 +1 shipit ! thx!
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user andrewpalumbo opened a pull request:

          https://github.com/apache/mahout/pull/293

          MAHOUT-1939 Revert for CLI Driver Fix

          fastutils shaded jar was creating an unwanted dependency-reduced-jar in mahout math. and not being passed to CLI or shell.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/andrewpalumbo/mahout MAHOUT-1939a

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/mahout/pull/293.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #293


          commit 96c3ad4df614ad31004e5c3cd6e115245c88fc44
          Author: Andrew Palumbo <apalumbo@apache.org>
          Date: 2017-03-09T15:38:34Z

          MAHOUT-1939 Revert for CLI Driver Fix


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user andrewpalumbo opened a pull request: https://github.com/apache/mahout/pull/293 MAHOUT-1939 Revert for CLI Driver Fix fastutils shaded jar was creating an unwanted dependency-reduced-jar in mahout math. and not being passed to CLI or shell. You can merge this pull request into a Git repository by running: $ git pull https://github.com/andrewpalumbo/mahout MAHOUT-1939 a Alternatively you can review and apply these changes as the patch at: https://github.com/apache/mahout/pull/293.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #293 commit 96c3ad4df614ad31004e5c3cd6e115245c88fc44 Author: Andrew Palumbo <apalumbo@apache.org> Date: 2017-03-09T15:38:34Z MAHOUT-1939 Revert for CLI Driver Fix
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user andrewpalumbo commented on the issue:

          https://github.com/apache/mahout/pull/293

          All tests pass

          Show
          githubbot ASF GitHub Bot added a comment - Github user andrewpalumbo commented on the issue: https://github.com/apache/mahout/pull/293 All tests pass
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/mahout/pull/293

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/mahout/pull/293
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Mahout-Quality #3445 (See https://builds.apache.org/job/Mahout-Quality/3445/)
          MAHOUT-1939 Remove fastutils shaded jar. Not being picked up by CLI (apalumbo: rev b32b6d9731481f00ea3b96e3129b9bc96e34eff5)

          • (edit) math/pom.xml
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Mahout-Quality #3445 (See https://builds.apache.org/job/Mahout-Quality/3445/ ) MAHOUT-1939 Remove fastutils shaded jar. Not being picked up by CLI (apalumbo: rev b32b6d9731481f00ea3b96e3129b9bc96e34eff5) (edit) math/pom.xml
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user andrewpalumbo commented on the issue:

          https://github.com/apache/mahout/pull/293

          Yeah I should have deleted.. at the last moment I decided to leave them in in case we needed them somewhere else.

          Sent from my Verizon Wireless 4G LTE smartphone

          -------- Original message --------
          From: Andrew Musselman <notifications@github.com>
          Date: 03/09/2017 7:43 AM (GMT-08:00)
          To: apache/mahout <mahout@noreply.github.com>
          Cc: Andrew Palumbo <ap.dev@outlook.com>, Author <author@noreply.github.com>
          Subject: Re: [apache/mahout] MAHOUT-1939 Revert for CLI Driver Fix (#293)

          @andrewmusselman commented on this pull request.

          Cool, can we delete this instead of commenting?


          You are receiving this because you authored the thread.
          Reply to this email directly, view it on GitHub<https://github.com/apache/mahout/pull/293#pullrequestreview-26061794>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AHU2HZH_3DYx8TsUqfCvY7GjW9HB1Ulqks5rkB4ogaJpZM4MYP3f>.

          Show
          githubbot ASF GitHub Bot added a comment - Github user andrewpalumbo commented on the issue: https://github.com/apache/mahout/pull/293 Yeah I should have deleted.. at the last moment I decided to leave them in in case we needed them somewhere else. Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Andrew Musselman <notifications@github.com> Date: 03/09/2017 7:43 AM (GMT-08:00) To: apache/mahout <mahout@noreply.github.com> Cc: Andrew Palumbo <ap.dev@outlook.com>, Author <author@noreply.github.com> Subject: Re: [apache/mahout] MAHOUT-1939 Revert for CLI Driver Fix (#293) @andrewmusselman commented on this pull request. Cool, can we delete this instead of commenting? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub< https://github.com/apache/mahout/pull/293#pullrequestreview-26061794 >, or mute the thread< https://github.com/notifications/unsubscribe-auth/AHU2HZH_3DYx8TsUqfCvY7GjW9HB1Ulqks5rkB4ogaJpZM4MYP3f >.

            People

            • Assignee:
              Andrew_Palumbo Andrew Palumbo
              Reporter:
              dlyubimov Dmitriy Lyubimov
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development

                  Agile