Details

    • Type: Bug
    • Status: Open
    • Priority: Blocker
    • Resolution: Unresolved
    • Affects Version/s: 0.13.1
    • Fix Version/s: 0.13.1
    • Component/s: build
    • Labels:
      None
    • Environment:

      Creating a project from maven built Mahout using sbt. Made critical since it seems to block using Mahout with sbt. At least I have found no way to do it.

      Description

      The maven repo is built with scala 2.10 always in the parent pom's

      {scala.compat.version}

      even when you only ask for Scala 2.11, this leads to the 2.11 jars never being found.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user rawkintrevo commented on the issue:

          https://github.com/apache/mahout/pull/331

          per MAHOUT-2020 this method wasn't working with SBT. So I have been refactoring.

          Note, now `mvn clean install` builds all sparks. `mvn clean -Pspark1.6` will result in quicker build times and only build spark 1.6, this is probably required so make the console work.

          Show
          githubbot ASF GitHub Bot added a comment - Github user rawkintrevo commented on the issue: https://github.com/apache/mahout/pull/331 per MAHOUT-2020 this method wasn't working with SBT. So I have been refactoring. Note, now `mvn clean install` builds all sparks. `mvn clean -Pspark1.6` will result in quicker build times and only build spark 1.6, this is probably required so make the console work.
          Hide
          rawkintrevo Trevor Grant added a comment -

          Pat Ferrel check out https://github.com/apache/mahout/pull/331

          Building that with `mvn clean install -DskipTests` (This purposefully builds spark1.6,spark2.0, and spark 2.1) I am able to make this silly empty sbt turn over.

          Note, sbt doesn't seem to be intelligent enough to check the local maven repo out of the box- and so one must add it.

          ```

          name := "mahout-build-with-sbt"

          version := "1.0"

          scalaVersion := "2.10.6"

          val mahoutVersion = "0.13.1-SNAPSHOT"
          val sparkVersion = "1.6.3"
          val sparkCompatVersion = "1.6"
          val scalaCompatVersion = "2.10"

          resolvers = "Local Maven Repository" at "file://" Path.userHome.absolutePath+"/.m2/repository"

          libraryDependencies ++= Seq(

          "org.apache.mahout" % "mahout-math" % mahoutVersion,
          "org.apache.mahout" %% s"mahout-math-scala" % mahoutVersion,
          "org.apache.mahout" %% s"mahout-spark$

          {sparkCompatVersion}

          " % mahoutVersion ,
          "org.apache.spark" %% s"spark-core" % sparkVersion
          )
          ```

          Show
          rawkintrevo Trevor Grant added a comment - Pat Ferrel check out https://github.com/apache/mahout/pull/331 Building that with `mvn clean install -DskipTests` (This purposefully builds spark1.6,spark2.0, and spark 2.1) I am able to make this silly empty sbt turn over. Note, sbt doesn't seem to be intelligent enough to check the local maven repo out of the box- and so one must add it. ``` name := "mahout-build-with-sbt" version := "1.0" scalaVersion := "2.10.6" val mahoutVersion = "0.13.1-SNAPSHOT" val sparkVersion = "1.6.3" val sparkCompatVersion = "1.6" val scalaCompatVersion = "2.10" resolvers = "Local Maven Repository" at "file://" Path.userHome.absolutePath+"/.m2/repository" libraryDependencies ++= Seq( "org.apache.mahout" % "mahout-math" % mahoutVersion, "org.apache.mahout" %% s"mahout-math-scala" % mahoutVersion, "org.apache.mahout" %% s"mahout-spark$ {sparkCompatVersion} " % mahoutVersion , "org.apache.spark" %% s"spark-core" % sparkVersion ) ```
          Hide
          pferrel Pat Ferrel added a comment - - edited

          Nothing to do with SBT, look in the parent POM, it always has

          {scala.combat.version}

          = 2.10 the rest of the child poms inherit that even if they were built for scala 2.11

          Show
          pferrel Pat Ferrel added a comment - - edited Nothing to do with SBT, look in the parent POM, it always has {scala.combat.version} = 2.10 the rest of the child poms inherit that even if they were built for scala 2.11
          Hide
          rawkintrevo Trevor Grant added a comment -

          Ok-

          (1) the Multi-Artifact biuld hasn't merged yet.
          (2) Scala 2.11 absolutely builds, and builds scala 2.11

          `mvn clean install -Pscala-2.11`

          That isn't to say sbt doesn't know what its doing. I never tested with that build tool. But you can build a scala-2.11 app and submit to SPark 2.1 cluster with scala 2.11 with no prob, and I can verify it is picking up scala 2.11, because if you build with scala 2.10 and submit that to a flink-2.11 cluster or vice verse it all goes haywire.

          Will test your issues/try to learn sbt to reproduce.

          Show
          rawkintrevo Trevor Grant added a comment - Ok- (1) the Multi-Artifact biuld hasn't merged yet. (2) Scala 2.11 absolutely builds, and builds scala 2.11 `mvn clean install -Pscala-2.11` That isn't to say sbt doesn't know what its doing. I never tested with that build tool. But you can build a scala-2.11 app and submit to SPark 2.1 cluster with scala 2.11 with no prob, and I can verify it is picking up scala 2.11, because if you build with scala 2.10 and submit that to a flink-2.11 cluster or vice verse it all goes haywire. Will test your issues/try to learn sbt to reproduce.
          Hide
          pferrel Pat Ferrel added a comment - - edited

          my script for creating a repo that is run in the Mahout build directory:

          mvn deploy:deploy-file -Durl=file:///Users/pat/.custom-scala-m2/repo/ -Dfile=//Users/pat/mahout/hdfs/target/mahout-hdfs-0.13.0.jar -DgroupId=org.apache.mahout -DartifactId=mahout-hdfs -Dversion=0.13.0
          mvn deploy:deploy-file -Durl=file:///Users/pat/.custom-scala-m2/repo/ -Dfile=//Users/pat/mahout/math/target/mahout-math-0.13.0.jar -DgroupId=org.apache.mahout -DartifactId=mahout-math -Dversion=0.13.0
          mvn deploy:deploy-file -Durl=file:///Users/pat/.custom-scala-m2/repo/ -Dfile=//Users/pat/mahout/math-scala/target/mahout-math-scala_2.11-0.13.0.jar -DgroupId=org.apache.mahout -DartifactId=mahout-math-scala_2.11 -Dversion=0.13.0
          mvn deploy:deploy-file -Durl=file:///Users/pat/.custom-scala-m2/repo/ -Dfile=//Users/pat/mahout/spark/target/mahout-spark_2.11-0.13.0.jar -DgroupId=org.apache.mahout -DartifactId=mahout-spark_2.11 -Dversion=0.13.0

          then in the build.sbt I get the artifacts using:

          resolvers += "Local Repository" at "file:///Users/pat/.custom-scala-m2/repo"

          Show
          pferrel Pat Ferrel added a comment - - edited my script for creating a repo that is run in the Mahout build directory: mvn deploy:deploy-file -Durl= file:///Users/pat/.custom-scala-m2/repo/ -Dfile=//Users/pat/mahout/hdfs/target/mahout-hdfs-0.13.0.jar -DgroupId=org.apache.mahout -DartifactId=mahout-hdfs -Dversion=0.13.0 mvn deploy:deploy-file -Durl= file:///Users/pat/.custom-scala-m2/repo/ -Dfile=//Users/pat/mahout/math/target/mahout-math-0.13.0.jar -DgroupId=org.apache.mahout -DartifactId=mahout-math -Dversion=0.13.0 mvn deploy:deploy-file -Durl= file:///Users/pat/.custom-scala-m2/repo/ -Dfile=//Users/pat/mahout/math-scala/target/mahout-math-scala_2.11-0.13.0.jar -DgroupId=org.apache.mahout -DartifactId=mahout-math-scala_2.11 -Dversion=0.13.0 mvn deploy:deploy-file -Durl= file:///Users/pat/.custom-scala-m2/repo/ -Dfile=//Users/pat/mahout/spark/target/mahout-spark_2.11-0.13.0.jar -DgroupId=org.apache.mahout -DartifactId=mahout-spark_2.11 -Dversion=0.13.0 then in the build.sbt I get the artifacts using: resolvers += "Local Repository" at "file:///Users/pat/.custom-scala-m2/repo"
          Hide
          pferrel Pat Ferrel added a comment -

          The problem here is with the multi-artifact build. The last version I tried always builds scala 2.10 version(s) of modules and makes the

          {scala.compat.version} = 2.10 in the master POM of the mvn repo. This trickles down to the child POMs in the repo and therefore the math-scala_2.11 is not found because the POM is telling the resolver to look for the scala 2.10 jar.

          This is very subtle because the jars are there for scala 2.10 and 2.11 but their poms use the master pom's {scala.compat.version}

          which seem to be always set to 2.10.

          The solution may be to not set the

          {scala.compat.version} in the master pom but set it in each scala module's pom with the correct scala version. Whatever the solution is must account for the possible use of more than one {scala.compat.version}

          that gets applied where the jar is stored in the maven directory structure.

          I know this works because I have created by hand, using `mvn deploy`, a maven type repo that excludes the master pom and things are resolved correctly. This seems to mean the

          {scala.compat.version}

          is set somewhere outside of the master pom, not sure where.

          Show
          pferrel Pat Ferrel added a comment - The problem here is with the multi-artifact build. The last version I tried always builds scala 2.10 version(s) of modules and makes the {scala.compat.version} = 2.10 in the master POM of the mvn repo. This trickles down to the child POMs in the repo and therefore the math-scala_2.11 is not found because the POM is telling the resolver to look for the scala 2.10 jar. This is very subtle because the jars are there for scala 2.10 and 2.11 but their poms use the master pom's {scala.compat.version} which seem to be always set to 2.10. The solution may be to not set the {scala.compat.version} in the master pom but set it in each scala module's pom with the correct scala version. Whatever the solution is must account for the possible use of more than one {scala.compat.version} that gets applied where the jar is stored in the maven directory structure. I know this works because I have created by hand, using `mvn deploy`, a maven type repo that excludes the master pom and things are resolved correctly. This seems to mean the {scala.compat.version} is set somewhere outside of the master pom, not sure where.
          Hide
          pferrel Pat Ferrel added a comment - - edited

          This may be a non-issue. Trevor said in email:

          The spark is included via maven classifier-

          the sbt line should be

          libraryDependencies += "org.apache.mahout" % "mahout-spark_2.11" %
          "0.13.1-SNAPSHOT" classifier "spark_2.1"

          This is a red herring, the real issue is scala 2.11 never builds correctly because poms end up with 2.10 as the scala.compat.version

          Show
          pferrel Pat Ferrel added a comment - - edited This may be a non-issue. Trevor said in email: The spark is included via maven classifier- the sbt line should be libraryDependencies += "org.apache.mahout" % "mahout-spark_2.11" % "0.13.1-SNAPSHOT" classifier "spark_2.1" This is a red herring, the real issue is scala 2.11 never builds correctly because poms end up with 2.10 as the scala.compat.version

            People

            • Assignee:
              rawkintrevo Trevor Grant
              Reporter:
              pferrel Pat Ferrel
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:

                Development