Uploaded image for project: 'Mahout'
  1. Mahout
  2. MAHOUT-1939

fastutil version clash with spark distributions

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None

      Description

      Version difference in fast util breaks sparse algebra (specifically, RandomAccessSparseVector in assign, e.g., vec *= 5).

      observed version in CDH:

      file:/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.21/jars/fastutil-6.3.jar

      mahout uses 7.0.12

      java.lang.UnsupportedOperationException
      at it.unimi.dsi.fastutil.ints.AbstractInt2DoubleMap$BasicEntry.setValue(AbstractInt2DoubleMap.java:146)
      at org.apache.mahout.math.RandomAccessSparseVector$RandomAccessElement.set(RandomAccessSparseVector.java:235)
      at org.apache.mahout.math.VectorView$DecoratorElement.set(VectorView.java:181)
      at org.apache.mahout.math.AbstractVector.assign(AbstractVector.java:536)
      at org.apache.mahout.math.scalabindings.RLikeVectorOps.$div$eq(RLikeVectorOps.scala:45)
      ...

        Issue Links

          Activity

          Hide
          dlyubimov Dmitriy Lyubimov added a comment - - edited

          perhaps mahout should include fast-util in a shaded form in mahout-math or mahout-math-scala.
          like this (in mahout-math):

          <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-shade-plugin</artifactId>
          <version>3.0.0</version>
          <executions>
          <execution>
          <phase>package</phase>
          <goals>
          <goal>shade</goal>
          </goals>
          <configuration>
          <artifactSet>
          <includes>
          <include>it.unimi.dsi:fastutil</include>
          </includes>
          </artifactSet>
          <relocations>
          <relocation>
          <pattern>it.unimi.dsi.fastutil</pattern>
          <shadedPattern>shaded.it.unimi.dsi.fastutil</shadedPattern>
          </relocation>
          </relocations>
          </configuration>
          </execution>
          </executions>
          </plugin>

          Show
          dlyubimov Dmitriy Lyubimov added a comment - - edited perhaps mahout should include fast-util in a shaded form in mahout-math or mahout-math-scala. like this (in mahout-math): <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>3.0.0</version> <executions> <execution> <phase>package</phase> <goals> <goal>shade</goal> </goals> <configuration> <artifactSet> <includes> <include>it.unimi.dsi:fastutil</include> </includes> </artifactSet> <relocations> <relocation> <pattern>it.unimi.dsi.fastutil</pattern> <shadedPattern>shaded.it.unimi.dsi.fastutil</shadedPattern> </relocation> </relocations> </configuration> </execution> </executions> </plugin>
          Hide
          Andrew_Palumbo Andrew Palumbo added a comment -

          Dmitriy Lyubimov How many CDH Spark versions does this break? Is this a blocker for the upcoming release? Seems so.

          Show
          Andrew_Palumbo Andrew Palumbo added a comment - Dmitriy Lyubimov How many CDH Spark versions does this break? Is this a blocker for the upcoming release? Seems so.
          Hide
          Andrew_Palumbo Andrew Palumbo added a comment -

          Dmitriy Lyubimov I think i mad this a blocker do you agree? I will convert it to critical now, and back to a blocker if you believe that it is.

          Show
          Andrew_Palumbo Andrew Palumbo added a comment - Dmitriy Lyubimov I think i mad this a blocker do you agree? I will convert it to critical now, and back to a blocker if you believe that it is.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user andrewpalumbo opened a pull request:

          https://github.com/apache/mahout/pull/285

          MAHOUT-1939: Shade fastutil jar conflicictng with CDH Spark

          Reports of fastutils versions conflicting on CDH. Shade our fastutils in `mahout-math`.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/andrewpalumbo/mahout MAHOUT-1939

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/mahout/pull/285.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #285


          commit 4095c32e99b69f48ff5e091d612eb65569c25f73
          Author: Andrew Palumbo <apalumbo@apache.org>
          Date: 2017-02-26T07:25:35Z

          MAHOUT-1939: Shade fastutil jar conflicictng with CDH


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user andrewpalumbo opened a pull request: https://github.com/apache/mahout/pull/285 MAHOUT-1939 : Shade fastutil jar conflicictng with CDH Spark Reports of fastutils versions conflicting on CDH. Shade our fastutils in `mahout-math`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/andrewpalumbo/mahout MAHOUT-1939 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/mahout/pull/285.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #285 commit 4095c32e99b69f48ff5e091d612eb65569c25f73 Author: Andrew Palumbo <apalumbo@apache.org> Date: 2017-02-26T07:25:35Z MAHOUT-1939 : Shade fastutil jar conflicictng with CDH
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user andrewpalumbo commented on the issue:

          https://github.com/apache/mahout/pull/285

          local tests pass on this. @dlyubimov could you please test this against the CDH version you found a conflict with?

          Show
          githubbot ASF GitHub Bot added a comment - Github user andrewpalumbo commented on the issue: https://github.com/apache/mahout/pull/285 local tests pass on this. @dlyubimov could you please test this against the CDH version you found a conflict with?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user andrewpalumbo commented on the issue:

          https://github.com/apache/mahout/pull/285

          committed in 4d1464ef4d7a1d278c828d29a906e8f431a2fcb5

          Show
          githubbot ASF GitHub Bot added a comment - Github user andrewpalumbo commented on the issue: https://github.com/apache/mahout/pull/285 committed in 4d1464ef4d7a1d278c828d29a906e8f431a2fcb5
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user andrewpalumbo closed the pull request at:

          https://github.com/apache/mahout/pull/285

          Show
          githubbot ASF GitHub Bot added a comment - Github user andrewpalumbo closed the pull request at: https://github.com/apache/mahout/pull/285
          Hide
          Andrew_Palumbo Andrew Palumbo added a comment -

          committed in 4d1464ef4d7a1d278c828d29a906e8f431a2fcb5

          Show
          Andrew_Palumbo Andrew Palumbo added a comment - committed in 4d1464ef4d7a1d278c828d29a906e8f431a2fcb5
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Jenkins build Mahout-Quality #3428 (See https://builds.apache.org/job/Mahout-Quality/3428/)
          MAHOUT-1939: Shade fastutil jar conflicictng with CDH Spark #285 (apalumbo: rev 4d1464ef4d7a1d278c828d29a906e8f431a2fcb5)

          • (edit) math/pom.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Jenkins build Mahout-Quality #3428 (See https://builds.apache.org/job/Mahout-Quality/3428/ ) MAHOUT-1939 : Shade fastutil jar conflicictng with CDH Spark #285 (apalumbo: rev 4d1464ef4d7a1d278c828d29a906e8f431a2fcb5) (edit) math/pom.xml
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user andrewpalumbo commented on the issue:

          https://github.com/apache/mahout/pull/285

          +1 shipit ! thx!

          Show
          githubbot ASF GitHub Bot added a comment - Github user andrewpalumbo commented on the issue: https://github.com/apache/mahout/pull/285 +1 shipit ! thx!

            People

            • Assignee:
              Andrew_Palumbo Andrew Palumbo
              Reporter:
              dlyubimov Dmitriy Lyubimov
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development

                  Agile