Kafka
  1. Kafka
  2. KAFKA-974

can't use public release maven repo because of failure of downloaded dependency

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:
      None

      Description

      trying to use the 0.8.0-beta1 release from public maven

      name := "Stub"

      version := "1.0.0.0"

      scalaVersion := "2.9.2"

      mainClass := Some("Stub")

      libraryDependencies ++= Seq(
      "org.apache.kafka" % "kafka_2.9.2" % "0.8.0-beta1"
      )

      results in

      Joes-MacBook-Air:stub joestein$ ./sbt compile
      [info] Set current project to default-63d5f2 (in build file:/opt/medialets/SymanticManager/scala/stub/)
      [info] Updating

      {file:/opt/medialets/SymanticManager/scala/stub/}

      default-63d5f2...
      [warn] [NOT FOUND ] javax.jms#jms;1.1!jms.jar (50ms)
      [warn] ==== public: tried
      [warn] http://repo1.maven.org/maven2/javax/jms/jms/1.1/jms-1.1.jar
      [warn] [NOT FOUND ] com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar (12ms)
      [warn] ==== public: tried
      [warn] http://repo1.maven.org/maven2/com/sun/jdmk/jmxtools/1.2.1/jmxtools-1.2.1.jar
      [warn] [NOT FOUND ] com.sun.jmx#jmxri;1.2.1!jmxri.jar (71ms)
      [warn] ==== public: tried
      [warn] http://repo1.maven.org/maven2/com/sun/jmx/jmxri/1.2.1/jmxri-1.2.1.jar
      [warn] ::::::::::::::::::::::::::::::::::::::::::::::
      [warn] :: FAILED DOWNLOADS ::
      [warn] :: ^ see resolution messages for details ^ ::
      [warn] ::::::::::::::::::::::::::::::::::::::::::::::
      [warn] :: javax.jms#jms;1.1!jms.jar
      [warn] :: com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar
      [warn] :: com.sun.jmx#jmxri;1.2.1!jmxri.jar
      [warn] ::::::::::::::::::::::::::::::::::::::::::::::
      [info]
      [warn] :: problems summary ::
      [warn] :::: WARNINGS
      [warn] [NOT FOUND ] javax.jms#jms;1.1!jms.jar (50ms)
      [warn] ==== public: tried
      [warn] http://repo1.maven.org/maven2/javax/jms/jms/1.1/jms-1.1.jar
      [warn] [NOT FOUND ] com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar (12ms)
      [warn] ==== public: tried
      [warn] http://repo1.maven.org/maven2/com/sun/jdmk/jmxtools/1.2.1/jmxtools-1.2.1.jar
      [warn] [NOT FOUND ] com.sun.jmx#jmxri;1.2.1!jmxri.jar (71ms)
      [warn] ==== public: tried
      [warn] http://repo1.maven.org/maven2/com/sun/jmx/jmxri/1.2.1/jmxri-1.2.1.jar
      [warn] ::::::::::::::::::::::::::::::::::::::::::::::
      [warn] :: FAILED DOWNLOADS ::
      [warn] :: ^ see resolution messages for details ^ ::
      [warn] ::::::::::::::::::::::::::::::::::::::::::::::
      [warn] :: javax.jms#jms;1.1!jms.jar
      [warn] :: com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar
      [warn] :: com.sun.jmx#jmxri;1.2.1!jmxri.jar
      [warn] ::::::::::::::::::::::::::::::::::::::::::::::
      [info]
      [info] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
      [error]

      {file:/opt/medialets/SymanticManager/scala/stub/}

      default-63d5f2/*:update: sbt.ResolveException: download failed: javax.jms#jms;1.1!jms.jar
      [error] download failed: com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar
      [error] download failed: com.sun.jmx#jmxri;1.2.1!jmxri.jar
      [error] Total time: 3 s, completed Jul 13, 2013 1:55:36 AM

      1. KAFKA-974.v2.patch
        0.6 kB
        Joe Stein
      2. KAFKA-974.patch
        0.8 kB
        Joe Stein

        Activity

        Hide
        Joe Stein added a comment -

        first patch that didn't work

        Show
        Joe Stein added a comment - first patch that didn't work
        Hide
        Joe Stein added a comment -

        tried this, same error

        diff --git a/project/Build.scala b/project/Build.scala
        index bad93db..c758178 100644
        — a/project/Build.scala
        +++ b/project/Build.scala
        @@ -39,7 +39,21 @@ object KafkaBuild extends Build {
        <url>http://www.apache.org/licenses/LICENSE-2.0.txt</url>
        <distribution>repo</distribution>
        </license>
        -</licenses>,
        +</licenses>
        +<dependencies>
        + <exclusion>
        + <groupId>com.sun.jmx</groupId>
        + <artifactId>jmxri</artifactId>
        + </exclusion>
        + <exclusion>
        + <groupId>com.sun.jdmk</groupId>
        + <artifactId>jmxtools</artifactId>
        + </exclusion>
        + <exclusion>
        + <groupId>javax.jms</groupId>
        + <artifactId>jms</artifactId>
        + </exclusion>
        +</dependencies>,
        scalacOptions ++= Seq("-deprecation", "-unchecked", "-g:none"),
        crossScalaVersions := Seq("2.8.0","2.8.2", "2.9.1", "2.9.2"),
        scalaVersion := "2.8.0",

        Show
        Joe Stein added a comment - tried this, same error diff --git a/project/Build.scala b/project/Build.scala index bad93db..c758178 100644 — a/project/Build.scala +++ b/project/Build.scala @@ -39,7 +39,21 @@ object KafkaBuild extends Build { <url> http://www.apache.org/licenses/LICENSE-2.0.txt </url> <distribution>repo</distribution> </license> -</licenses>, +</licenses> +<dependencies> + <exclusion> + <groupId>com.sun.jmx</groupId> + <artifactId>jmxri</artifactId> + </exclusion> + <exclusion> + <groupId>com.sun.jdmk</groupId> + <artifactId>jmxtools</artifactId> + </exclusion> + <exclusion> + <groupId>javax.jms</groupId> + <artifactId>jms</artifactId> + </exclusion> +</dependencies>, scalacOptions ++= Seq("-deprecation", "-unchecked", "-g:none"), crossScalaVersions := Seq("2.8.0","2.8.2", "2.9.1", "2.9.2"), scalaVersion := "2.8.0",
        Hide
        Joe Stein added a comment -

        tried this with no luck, same error

        diff --git a/project/Build.scala b/project/Build.scala
        index bad93db..219d7e0 100644
        — a/project/Build.scala
        +++ b/project/Build.scala
        @@ -52,7 +52,11 @@ object KafkaBuild extends Build {
        javacOptions ++= Seq("-Xlint:unchecked", "-source", "1.5"),
        parallelExecution in Test := false, // Prevent tests from overrunning each other
        libraryDependencies ++= Seq(

        • "log4j" % "log4j" % "1.2.15",
          + "log4j" % "log4j" % "1.2.15" excludeAll(
          + ExclusionRule(organization = "com.sun.jdmk"),
          + ExclusionRule(organization = "com.sun.jmx"),
          + ExclusionRule(organization = "javax.jms")
          + ),
          "net.sf.jopt-simple" % "jopt-simple" % "3.2",
          "org.slf4j" % "slf4j-simple" % "1.6.4"
          ),
        Show
        Joe Stein added a comment - tried this with no luck, same error diff --git a/project/Build.scala b/project/Build.scala index bad93db..219d7e0 100644 — a/project/Build.scala +++ b/project/Build.scala @@ -52,7 +52,11 @@ object KafkaBuild extends Build { javacOptions ++= Seq("-Xlint:unchecked", "-source", "1.5"), parallelExecution in Test := false, // Prevent tests from overrunning each other libraryDependencies ++= Seq( "log4j" % "log4j" % "1.2.15", + "log4j" % "log4j" % "1.2.15" excludeAll( + ExclusionRule(organization = "com.sun.jdmk"), + ExclusionRule(organization = "com.sun.jmx"), + ExclusionRule(organization = "javax.jms") + ), "net.sf.jopt-simple" % "jopt-simple" % "3.2", "org.slf4j" % "slf4j-simple" % "1.6.4" ),
        Hide
        Joe Stein added a comment -

        well, as far as SBT goes http://www.scala-sbt.org/release/docs/Detailed-Topics/Library-Management#exclude-transitive-dependencies it looks like exclude is the right command to use not excludeAll

        diff --git a/project/Build.scala b/project/Build.scala
        index bad93db..b3858f3 100644
        — a/project/Build.scala
        +++ b/project/Build.scala
        @@ -52,7 +52,7 @@ object KafkaBuild extends Build {
        javacOptions ++= Seq("-Xlint:unchecked", "-source", "1.5"),
        parallelExecution in Test := false, // Prevent tests from overrunning each other
        libraryDependencies ++= Seq(

        • "log4j" % "log4j" % "1.2.15",
          + "log4j" % "log4j" % "1.2.15" exclude("javax.jms", "jms"),
          "net.sf.jopt-simple" % "jopt-simple" % "3.2",
          "org.slf4j" % "slf4j-simple" % "1.6.4"
          ),

        however, still same error though

        [error]

        {file:/opt/medialets/SymanticManager/scala/stub/}

        default-63d5f2/*:update: sbt.ResolveException: download failed: javax.jms#jms;1.1!jms.jar

        the pom does look better with the exclusion in the place I expected

        <dependency>
        <groupId>log4j</groupId>
        <artifactId>log4j</artifactId>
        <version>1.2.15</version>
        <exclusions>
        <exclusion>
        <groupId>javax.jms</groupId>
        <artifactId>jms</artifactId>
        </exclusion>
        </exclusions>
        </dependency>

        still not working

        Show
        Joe Stein added a comment - well, as far as SBT goes http://www.scala-sbt.org/release/docs/Detailed-Topics/Library-Management#exclude-transitive-dependencies it looks like exclude is the right command to use not excludeAll diff --git a/project/Build.scala b/project/Build.scala index bad93db..b3858f3 100644 — a/project/Build.scala +++ b/project/Build.scala @@ -52,7 +52,7 @@ object KafkaBuild extends Build { javacOptions ++= Seq("-Xlint:unchecked", "-source", "1.5"), parallelExecution in Test := false, // Prevent tests from overrunning each other libraryDependencies ++= Seq( "log4j" % "log4j" % "1.2.15", + "log4j" % "log4j" % "1.2.15" exclude("javax.jms", "jms"), "net.sf.jopt-simple" % "jopt-simple" % "3.2", "org.slf4j" % "slf4j-simple" % "1.6.4" ), however, still same error though [error] {file:/opt/medialets/SymanticManager/scala/stub/} default-63d5f2/*:update: sbt.ResolveException: download failed: javax.jms#jms;1.1!jms.jar the pom does look better with the exclusion in the place I expected <dependency> <groupId>log4j</groupId> <artifactId>log4j</artifactId> <version>1.2.15</version> <exclusions> <exclusion> <groupId>javax.jms</groupId> <artifactId>jms</artifactId> </exclusion> </exclusions> </dependency> still not working
        Hide
        Joe Stein added a comment -

        That last fix did work and attached patch

        to use public maven

        "org.apache.kafka" % "kafka_2.9.2" % "0.8.0-beta1" intransitive()

        the key in this last patch was to include intransitive() so that SBT knows to not fetch the dependencies we specified in the pom http://www.scala-sbt.org/release/docs/Detailed-Topics/Library-Management#disable-transitivity

        Show
        Joe Stein added a comment - That last fix did work and attached patch to use public maven "org.apache.kafka" % "kafka_2.9.2" % "0.8.0-beta1" intransitive() the key in this last patch was to include intransitive() so that SBT knows to not fetch the dependencies we specified in the pom http://www.scala-sbt.org/release/docs/Detailed-Topics/Library-Management#disable-transitivity
        Hide
        Jay Kreps added a comment -

        w00t!

        Show
        Jay Kreps added a comment - w00t!
        Hide
        Chris Riccomini added a comment -

        Hey Joe,

        This looks really good. Per our discussion on the mailing list, I'm updating with some issues I've found:

        1. Maven central can't resolve it properly (POM is different from Apache release). Have to use Apache release repo directly to get things to work.
        2. Exclusions must be manually applied even though they exist in Kafka's POM already. I think Maven can handle this automatically, if the POM is done right.
        3. Weird parent block in Kafka POMs that points to org.apache.
        4. Would be nice to publish kafka-test jars as well.
        5. Would be nice to have SNAPSHOT releases off of trunk using a Hudson job.

        Our hypothesis regarding the first issue is that it was caused by duplicate publishing during testing, and it should go away in the future.

        Regarding number 2, I have to explicitly exclude the following when depending on Kafka:

        exclude module: 'jms'
        exclude module: 'jmxtools'
        exclude module: 'jmxri'

        I believe these just need to be excluded from the appropriate jars in the actual SBT build file, to fix this issue. I see JMS is excluded from ZK, but it's probably being pulled in from somewhere else, anyway.

        Regarding number 3, it is indeed listed as something to do on the Apache publication page (http://www.apache.org/dev/publishing-maven-artifacts.html). I can't find an example of anyone using it, but it doesn't seem to be doing any harm.

        Also, regarding your intransitive() call, that is disabling ALL dependencies not just the exclusions, I believe. I think that the "proper" way to do that would be to do what I've done: exclude("jms", "jmxtools", "jmxri"). Regardless, fixing number 2, above, should mean that intransitive()/exclude() are not required.

        Show
        Chris Riccomini added a comment - Hey Joe, This looks really good. Per our discussion on the mailing list, I'm updating with some issues I've found: 1. Maven central can't resolve it properly (POM is different from Apache release). Have to use Apache release repo directly to get things to work. 2. Exclusions must be manually applied even though they exist in Kafka's POM already. I think Maven can handle this automatically, if the POM is done right. 3. Weird parent block in Kafka POMs that points to org.apache. 4. Would be nice to publish kafka-test jars as well. 5. Would be nice to have SNAPSHOT releases off of trunk using a Hudson job. Our hypothesis regarding the first issue is that it was caused by duplicate publishing during testing, and it should go away in the future. Regarding number 2, I have to explicitly exclude the following when depending on Kafka: exclude module: 'jms' exclude module: 'jmxtools' exclude module: 'jmxri' I believe these just need to be excluded from the appropriate jars in the actual SBT build file, to fix this issue. I see JMS is excluded from ZK, but it's probably being pulled in from somewhere else, anyway. Regarding number 3, it is indeed listed as something to do on the Apache publication page ( http://www.apache.org/dev/publishing-maven-artifacts.html ). I can't find an example of anyone using it, but it doesn't seem to be doing any harm. Also, regarding your intransitive() call, that is disabling ALL dependencies not just the exclusions, I believe. I think that the "proper" way to do that would be to do what I've done: exclude("jms", "jmxtools", "jmxri"). Regardless, fixing number 2, above, should mean that intransitive()/exclude() are not required.
        Hide
        Joe Stein added a comment -

        resolving because we need another release to take the patch and publish it since it is working in the apache repository fine

        Show
        Joe Stein added a comment - resolving because we need another release to take the patch and publish it since it is working in the apache repository fine
        Hide
        Michael Noll added a comment -

        FWIW here is an sbt snippet that resolves this problem. Chris' last comment was pointing me in the right direction but I still had to figure out the correct sbt syntax – literally using exclude("jms", "jmxtools", "jmxri") did not work.

        libraryDependencies ++= Seq(
          "org.apache.kafka" % "kafka_2.10" % "0.8.0"
            exclude("javax.jms", "jms")
            exclude("com.sun.jdmk", "jmxtools")
            exclude("com.sun.jmx", "jmxri"),
          // Alternatively, this apparently also works but it will exclude ALL deps of the excluded organizations:
          //"org.apache.kafka" % "kafka_2.10" % "0.8.0" excludeAll(
          //  ExclusionRule(organization = "com.sun.jdmk"),
          //  ExclusionRule(organization = "com.sun.jmx"),
          //  ExclusionRule(organization = "javax.jms")
          //),
          ...other dependencies...
        )
        

        Versions:

        • sbt 0.13.0
        • Scala 2.10.3
        Show
        Michael Noll added a comment - FWIW here is an sbt snippet that resolves this problem. Chris' last comment was pointing me in the right direction but I still had to figure out the correct sbt syntax – literally using exclude("jms", "jmxtools", "jmxri") did not work. libraryDependencies ++= Seq( "org.apache.kafka" % "kafka_2.10" % "0.8.0" exclude( "javax.jms" , "jms" ) exclude( "com.sun.jdmk" , "jmxtools" ) exclude( "com.sun.jmx" , "jmxri" ), // Alternatively, this apparently also works but it will exclude ALL deps of the excluded organizations: // "org.apache.kafka" % "kafka_2.10" % "0.8.0" excludeAll( // ExclusionRule(organization = "com.sun.jdmk" ), // ExclusionRule(organization = "com.sun.jmx" ), // ExclusionRule(organization = "javax.jms" ) //), ...other dependencies... ) Versions: sbt 0.13.0 Scala 2.10.3

          People

          • Assignee:
            Unassigned
            Reporter:
            Joe Stein
          • Votes:
            2 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development