Kafka
  1. Kafka
  2. KAFKA-133

Publish kafka jar to a public maven repository

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.6, 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:

      Description

      The released kafka jar must be download manually and then deploy to a private repository before they can be used by a developer using maven2.

      Similar to other Apache projects, it will be nice to have a way to publish Kafka releases to a public maven repo.

      In the past, we gave it a try using sbt publish to Sonatype Nexus maven repo, but ran into some authentication problems. It will be good to revisit this and get it resolved.

      1. pom.xml
        4 kB
        Thomas Dudziak
      2. KAFKA-133.patch
        4 kB
        Maxime Brugidou

        Issue Links

          Activity

          Hide
          Russell Jurney added a comment -

          When I type: ./sbt actions, I see one to create a POM.xml

          We're having to build kafka and host it on S3 ourselves, which is unfortunate.

          Show
          Russell Jurney added a comment - When I type: ./sbt actions, I see one to create a POM.xml We're having to build kafka and host it on S3 ourselves, which is unfortunate.
          Hide
          Alex Gray added a comment -

          Thanks Scott, for the links. Just searching for "sbt s3" does not get too many hits, but the best one by far is:
          http://stackoverflow.com/questions/9958103/how-to-publish-to-amazon-s3-with-sbt
          So far, from what I can tell, either copy the ivy jar and do the following (the million dollar question is how to configure the setters):

          { val s3r = new org.springframework.aws.ivy.S3Resolver // call its setters to configure it, see api docs above etc. publishTo := Some(new sbt.RawRepository(s3r) }

          or use sbt-s3, which I haven't had any success in.
          Nonetheless, I think I have enough to experiment with.
          Thanks!

          Show
          Alex Gray added a comment - Thanks Scott, for the links. Just searching for "sbt s3" does not get too many hits, but the best one by far is: http://stackoverflow.com/questions/9958103/how-to-publish-to-amazon-s3-with-sbt So far, from what I can tell, either copy the ivy jar and do the following (the million dollar question is how to configure the setters): { val s3r = new org.springframework.aws.ivy.S3Resolver // call its setters to configure it, see api docs above etc. publishTo := Some(new sbt.RawRepository(s3r) } or use sbt-s3, which I haven't had any success in. Nonetheless, I think I have enough to experiment with. Thanks!
          Hide
          Scott Carey added a comment -

          The build is using SBT, whic is using Ivy behind the scenes, so there is no pom.xml. One is generated during the build and can be published to a maven repo if SBT is configured correctly. I'm not familiar with how to do that.

          The best I can find quickly with a search are:
          https://groups.google.com/forum/?fromgroups=#!topic/simple-build-tool/CJS8GvXO4j0
          https://groups.google.com/forum/?fromgroups=#!topic/simple-build-tool/HaEv4P9Mxkw

          Show
          Scott Carey added a comment - The build is using SBT, whic is using Ivy behind the scenes, so there is no pom.xml. One is generated during the build and can be published to a maven repo if SBT is configured correctly. I'm not familiar with how to do that. The best I can find quickly with a search are: https://groups.google.com/forum/?fromgroups=#!topic/simple-build-tool/CJS8GvXO4j0 https://groups.google.com/forum/?fromgroups=#!topic/simple-build-tool/HaEv4P9Mxkw
          Hide
          Jun Rao added a comment -

          Have you tried the following as listed in earlier comments?

          ./sbt publish-local && ./sbt publish

          Show
          Jun Rao added a comment - Have you tried the following as listed in earlier comments? ./sbt publish-local && ./sbt publish
          Hide
          Alex Gray added a comment -

          Silly question, but I just downloaded 0.8 branch and I see no pom.xml.
          I would like to locally build kafka and publish to our private local maven repo.
          Is this currently possible? Do I have to do something for the pom.xml to get generated.
          Sorry for the newbie question.

          I noticed that when I run ./sbt publish-local, the artifacts get pushed to my local repo:
          ~/.ivy2/local/org.apache/contrib_2.8.0
          ~/.ivy2/local/org.apache/kafka_2.8.0

          I would like to publish to our private local maven repo (which just so happens to be in S3), but I'm used to modifying a pom.xml to add the required plugins/repoID's/etc/etc/, hence the question on where the pom.xml is.

          Thanks!

          Show
          Alex Gray added a comment - Silly question, but I just downloaded 0.8 branch and I see no pom.xml. I would like to locally build kafka and publish to our private local maven repo. Is this currently possible? Do I have to do something for the pom.xml to get generated. Sorry for the newbie question. I noticed that when I run ./sbt publish-local, the artifacts get pushed to my local repo: ~/.ivy2/local/org.apache/contrib_2.8.0 ~/.ivy2/local/org.apache/kafka_2.8.0 I would like to publish to our private local maven repo (which just so happens to be in S3), but I'm used to modifying a pom.xml to add the required plugins/repoID's/etc/etc/, hence the question on where the pom.xml is. Thanks!
          Hide
          Neha Narkhede added a comment -

          I filed KAFKA-826 to track the metrics 2.2.0 change. Patches are welcome !

          Show
          Neha Narkhede added a comment - I filed KAFKA-826 to track the metrics 2.2.0 change. Patches are welcome !
          Hide
          Darren Sargent added a comment -

          This ticket is marked as fixed, but the latest on the 0.8 branch still depends on an unpublished version of Yammer - this is from Build.scala:

          <dependency>
          <groupId>com.yammer.metrics</groupId>
          <artifactId>metrics-core</artifactId>
          <version>3.0.0-c0c8be71</version>
          <scope>compile</scope>
          </dependency>
          <dependency>
          <groupId>com.yammer.metrics</groupId>
          <artifactId>metrics-annotations</artifactId>
          <version>3.0.0-c0c8be71</version>
          <scope>compile</scope>
          </dependency>

          Maven Central doesn't have a version anything like the one mentioned above: http://repo2.maven.org/maven2/com/yammer/metrics/metrics-core/

          Show
          Darren Sargent added a comment - This ticket is marked as fixed, but the latest on the 0.8 branch still depends on an unpublished version of Yammer - this is from Build.scala: <dependency> <groupId>com.yammer.metrics</groupId> <artifactId>metrics-core</artifactId> <version>3.0.0-c0c8be71</version> <scope>compile</scope> </dependency> <dependency> <groupId>com.yammer.metrics</groupId> <artifactId>metrics-annotations</artifactId> <version>3.0.0-c0c8be71</version> <scope>compile</scope> </dependency> Maven Central doesn't have a version anything like the one mentioned above: http://repo2.maven.org/maven2/com/yammer/metrics/metrics-core/
          Hide
          Scott Carey added a comment -

          Additionally, 0.8.x can not be an official Apache release in this form because the source tarball would not compile or provide all of the tools necessary to build or reference publicly available libraries (i.e. the metrics snapshot jar is ephemeral).

          Show
          Scott Carey added a comment - Additionally, 0.8.x can not be an official Apache release in this form because the source tarball would not compile or provide all of the tools necessary to build or reference publicly available libraries (i.e. the metrics snapshot jar is ephemeral).
          Hide
          Scott Carey added a comment -

          0.8.x needs to depend on a released version of metrics.

          I currently use yammer-metrics in my code, and was excited to see that kafka was going to use metrics so that we could clean up and standardize things there. However, since it depends essentially on trunk (3.0.0-SNAPSHOT) including Kafka 0.8 breaks and there is no release plan or date for when 3.0.0 will even exist.

          Even worse, if we try and compile against the CURRENT yammer-metrics head 3.0.0-SNAPSHOT kafka fails to compile because it is ... trunk and things change.

          0.8.x needs to depend on a released version of metrics.

          Show
          Scott Carey added a comment - 0.8.x needs to depend on a released version of metrics. I currently use yammer-metrics in my code, and was excited to see that kafka was going to use metrics so that we could clean up and standardize things there. However, since it depends essentially on trunk (3.0.0-SNAPSHOT) including Kafka 0.8 breaks and there is no release plan or date for when 3.0.0 will even exist. Even worse, if we try and compile against the CURRENT yammer-metrics head 3.0.0-SNAPSHOT kafka fails to compile because it is ... trunk and things change. 0.8.x needs to depend on a released version of metrics.
          Hide
          Rajasekar Elango added a comment - - edited

          Are we expecting 0.7.3 release with feature to publish to maven repository..? If so do you know the time frame.. ? Also publish-local publish it to local ivy repository.. Is there a way to change it to publish to local maven repository..?

          Raja.

          Show
          Rajasekar Elango added a comment - - edited Are we expecting 0.7.3 release with feature to publish to maven repository..? If so do you know the time frame.. ? Also publish-local publish it to local ivy repository.. Is there a way to change it to publish to local maven repository..? Raja.
          Hide
          ben fleis added a comment -

          I'm new to maven, so this may be a dumb question: would it be reasonable/easy to publish nightly jars (via Apache?), under the 0.8.0-SNAPSHOT tag? I have an automated build system that currently git's and builds the whole thing, which I would love to replace with simple jar files.

          Show
          ben fleis added a comment - I'm new to maven, so this may be a dumb question: would it be reasonable/easy to publish nightly jars (via Apache?), under the 0.8.0-SNAPSHOT tag? I have an automated build system that currently git's and builds the whole thing, which I would love to replace with simple jar files.
          Hide
          Joe Stein added a comment -

          committed to 0.8 branch also

          Show
          Joe Stein added a comment - committed to 0.8 branch also
          Hide
          Joe Stein added a comment -

          << Just to clarify – this is marked as resolved, but I don't see any artifacts in maven central (search.maven.org). Is there some other repo to which the artifacts have been released?

          0.8 is not released yet. once it is the artifacts for 0.8 will be published

          Show
          Joe Stein added a comment - << Just to clarify – this is marked as resolved, but I don't see any artifacts in maven central (search.maven.org). Is there some other repo to which the artifacts have been released? 0.8 is not released yet. once it is the artifacts for 0.8 will be published
          Hide
          Evan Pollan added a comment -

          Just to clarify – this is marked as resolved, but I don't see any artifacts in maven central (search.maven.org). Is there some other repo to which the artifacts have been released?

          Show
          Evan Pollan added a comment - Just to clarify – this is marked as resolved, but I don't see any artifacts in maven central (search.maven.org). Is there some other repo to which the artifacts have been released?
          Hide
          Jun Rao added a comment -

          Joe, do you plan to merge the commit to the 0.8 branch?

          Show
          Jun Rao added a comment - Joe, do you plan to merge the commit to the 0.8 branch?
          Hide
          Joe Stein added a comment -

          I committed this with 2 minor changes

          1) removed the commented out line of code that is not needed anymore
          2) overrode the organization to be org.apache so the groupId shows up as such

          sbt works for using this locally now

          "org.apache" % "core-kafka_2.8.0" % "0.8.0-SNAPSHOT"

          it is a great step forward toward for helping folks to use the core code. thanks!

          any other issues that come up I will follow-up accordingly with them and will shepherd that we publish once released (since we are not there yet it is premature but saw no reason to not get these changes up in trunk and resolve this ticket)

          Show
          Joe Stein added a comment - I committed this with 2 minor changes 1) removed the commented out line of code that is not needed anymore 2) overrode the organization to be org.apache so the groupId shows up as such sbt works for using this locally now "org.apache" % "core-kafka_2.8.0" % "0.8.0-SNAPSHOT" it is a great step forward toward for helping folks to use the core code. thanks! any other issues that come up I will follow-up accordingly with them and will shepherd that we publish once released (since we are not there yet it is premature but saw no reason to not get these changes up in trunk and resolve this ticket)
          Hide
          Joe Stein added a comment -

          this looks really good so far I was able to get ./sbt publish-local to run just fine and integrate the result loca .iv2 with another sbt project I have using "kafka" % "core-kafka_2.8.0" % "0.8.0-SNAPSHOT"

          couple things

          1) the groupID should be = org.apache
          2) do you see any reason this would not run on the 0.7.2 branch? I am going to propose a 0.7.3 release with this change also in there (less the metrics as that is only in 0.8 and changes to version all minor details). I don't but figure I would ask

          notes for self: http://www.apache.org/dev/publishing-maven-artifacts.html

          Show
          Joe Stein added a comment - this looks really good so far I was able to get ./sbt publish-local to run just fine and integrate the result loca .iv2 with another sbt project I have using "kafka" % "core-kafka_2.8.0" % "0.8.0-SNAPSHOT" couple things 1) the groupID should be = org.apache 2) do you see any reason this would not run on the 0.7.2 branch? I am going to propose a 0.7.3 release with this change also in there (less the metrics as that is only in 0.8 and changes to version all minor details). I don't but figure I would ask notes for self: http://www.apache.org/dev/publishing-maven-artifacts.html
          Hide
          Maxime Brugidou added a comment -

          Taking Otis notes into account, Here is the simplest patch I could make, that applies cleanly on 0.8:

          Here are the changes:

          • Version is 0.8.0-SNAPSHOT until 0.8.0 becomes stable
          • You need to change the publish URL and credentials location
          • Use com.101tec.zkclient 0.2
          • Use com.yammer.metrics.metrics-* 3.0.0-SNAPSHOT (I had to use snapshot because there is no 3.0.0 stable)
          • Use org.slf4j.slf4j-simple 1.6.4 ("latest.release" was used)

          To publish:

          • ./sbt publish-local && ./sbt publish

          This will create a pom for kafka that depends on all the subprojects. The "main" project that you generally want to use is core-kafka_2.8.0 like this:

          <dependency>
          <groupId>kafka</groupId>
          <artifactId>core-kafka_2.8.0</artifactId>
          <version>0.8.0-SNAPSHOT</version>
          </dependency>

          Show
          Maxime Brugidou added a comment - Taking Otis notes into account, Here is the simplest patch I could make, that applies cleanly on 0.8: Here are the changes: Version is 0.8.0-SNAPSHOT until 0.8.0 becomes stable You need to change the publish URL and credentials location Use com.101tec.zkclient 0.2 Use com.yammer.metrics.metrics-* 3.0.0-SNAPSHOT (I had to use snapshot because there is no 3.0.0 stable) Use org.slf4j.slf4j-simple 1.6.4 ("latest.release" was used) To publish: ./sbt publish-local && ./sbt publish This will create a pom for kafka that depends on all the subprojects. The "main" project that you generally want to use is core-kafka_2.8.0 like this: <dependency> <groupId>kafka</groupId> <artifactId>core-kafka_2.8.0</artifactId> <version>0.8.0-SNAPSHOT</version> </dependency>
          Hide
          Neha Narkhede added a comment -

          That's great. Thanks a lot Otis for seeing this through. We will look into the process of publishing to Maven now.

          Show
          Neha Narkhede added a comment - That's great. Thanks a lot Otis for seeing this through. We will look into the process of publishing to Maven now.
          Hide
          Otis Gospodnetic added a comment -
          Show
          Otis Gospodnetic added a comment - There you have it: zkclient 0.2: http://search.maven.org/#artifactdetails%7Ccom.101tec%7Czkclient%7C0.2%7Cjar metrics 2.2.0: http://search.maven.org/#artifactdetails%7Ccom.yammer.metrics%7Cmetrics-parent%7C2.2.0%7Cpom This is now the most popular Kafka issue by far!
          Hide
          Jay Kreps added a comment -

          Hey Otis, that's awesome. Thanks for taking point on that and keeping the focus on this issue.

          Show
          Jay Kreps added a comment - Hey Otis, that's awesome. Thanks for taking point on that and keeping the focus on this issue.
          Hide
          Otis Gospodnetic added a comment -

          Update on the zklient front - it's getting Sonatyped: https://issues.sonatype.org/browse/OSSRH-4783 and Johannes Zillmann says it should be in there in 2-3 days.

          Show
          Otis Gospodnetic added a comment - Update on the zklient front - it's getting Sonatyped: https://issues.sonatype.org/browse/OSSRH-4783 and Johannes Zillmann says it should be in there in 2-3 days.
          Hide
          Otis Gospodnetic added a comment -

          Here is the procedure that one of Sematext engineers shared with the rest of our team when he set up one of our OSS projects in Sonatype's Maven repo (so ignore references to ActionGenerator):

          1. The first thing is to create an account at
          https://issues.sonatype.org/.

          2. Second thing is to create a ticket at
          https://issues.sonatype.org/browse/OSSRH with 'Project' type and
          subtype of 'Support - Open Source Project Repository Hosting' and
          issue type of 'New Project'. You need to fill all the information
          there and submit issue. After that you'll have to wait from a few
          hours to two days for it to be ready.

          3. After 2) is ready you need to install GPG client, create your key
          and publish it:

          • gpg --gen-key (to generate key)
          • gpg --list-keys (to list keys)
          • gpg --keyserver hkp://pool.sks-keyservers.net --send-keys YOUR_KEY_ID (to upload it)
          • gpg --keyserver hkp://pool.sks-keyservers.net --recv-keys YOUR_KEY_ID (to check if it was uploaded)

          4. Add GPG maven plugin to your project. The best way is to create a
          new profile, not to sign artifacts during local builds. With an
          profile like the one below, you use if by adding -DperformRelease=true
          to maven command. It will ask for your key password and sign artifacts
          with your key. This is needed to publish artifacts to maven
          repository. I've created the following profile:

          <profiles>
          <profile>
          <id>release-sign-artifacts</id>
          <activation>
          <property>
          <name>performRelease</name>
          <value>true</value>
          </property>
          </activation>
          <build>
          <plugins>
          <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-gpg-plugin</artifactId>
          <version>1.1</version>
          <executions>
          <execution>
          <id>sign-artifacts</id>
          <phase>verify</phase>
          <goals>
          <goal>sign</goal>
          </goals>
          </execution>
          </executions>
          </plugin>
          </plugins>
          </build>
          </profile>
          </profiles>

          5. You need to ensure that you have the following in your POM files:

          • <modelVersion>
          • <groupId>
          • <artifactId>
          • <version>
          • <packaging>
          • <name>
          • <description>
          • <url>
          • <licenses>
          • <scm><url>
          • <scm><connection>
          • <developers>

          6. Add Sonatype parent to your parent POM:
          <parent>
          <groupId>org.sonatype.oss</groupId>
          <artifactId>oss-parent</artifactId>
          <version>7</version>
          </parent>

          7. Add SCM information to your project. For example like this:
          <scm>
          <connection>scm:git:git@github.com:sematext/ActionGenerator.git</connection>
          <developerConnection>scm:git:git@github.com:sematext/ActionGenerator.git</developerConnection>
          <url>https://github.com/sematext/ActionGenerator</url>
          </scm>

          8. Modify your settings.xml file (you should have it in your maven
          home directory) and add the following:
          <servers>
          <server>
          <id>sonatype-nexus-snapshots</id>
          <username>your-jira-id</username>
          <password>your-jira-pwd</password>
          </server>
          <server>
          <id>sonatype-nexus-staging</id>
          <username>your-jira-id</username>
          <password>your-jira-pwd</password>
          </server>
          </servers>

          Where:

          • your-jira-id is the ID of your account created in step 1
          • your-jira-pwd is the password for account created in step 1

          9. Build your project, sign artifacts and deploy:
          $mvn clean install deploy -DperformRelease=true

          10. After that, your artifcts should be uloaded to staging sonatype.
          You now should login into https://oss.sonatype.org/ with the
          previously created account, choose repository and click "Close". If
          you did everything right, close will be successful.

          11. Now click "Release" on the closed repository and comment on the
          issue created in step 2) that you released the artifacts. Someone at
          Sonatype will review the artifacts and accept it. The review process
          will only happen once and the initial release. Later releases will be
          released automaticaly.

          This is what I did. Now I'm waiting for ActionGenerator to be
          released. In case you would like to go deeper into that process, there
          is a nice documentation available at
          https://docs.sonatype.org/display/Repository/Sonatype+OSS+Maven+Repository+Usage+Guide

          Show
          Otis Gospodnetic added a comment - Here is the procedure that one of Sematext engineers shared with the rest of our team when he set up one of our OSS projects in Sonatype's Maven repo (so ignore references to ActionGenerator): 1. The first thing is to create an account at https://issues.sonatype.org/ . 2. Second thing is to create a ticket at https://issues.sonatype.org/browse/OSSRH with 'Project' type and subtype of 'Support - Open Source Project Repository Hosting' and issue type of 'New Project'. You need to fill all the information there and submit issue. After that you'll have to wait from a few hours to two days for it to be ready. 3. After 2) is ready you need to install GPG client, create your key and publish it: gpg --gen-key (to generate key) gpg --list-keys (to list keys) gpg --keyserver hkp://pool.sks-keyservers.net --send-keys YOUR_KEY_ID (to upload it) gpg --keyserver hkp://pool.sks-keyservers.net --recv-keys YOUR_KEY_ID (to check if it was uploaded) 4. Add GPG maven plugin to your project. The best way is to create a new profile, not to sign artifacts during local builds. With an profile like the one below, you use if by adding -DperformRelease=true to maven command. It will ask for your key password and sign artifacts with your key. This is needed to publish artifacts to maven repository. I've created the following profile: <profiles> <profile> <id>release-sign-artifacts</id> <activation> <property> <name>performRelease</name> <value>true</value> </property> </activation> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-gpg-plugin</artifactId> <version>1.1</version> <executions> <execution> <id>sign-artifacts</id> <phase>verify</phase> <goals> <goal>sign</goal> </goals> </execution> </executions> </plugin> </plugins> </build> </profile> </profiles> 5. You need to ensure that you have the following in your POM files: <modelVersion> <groupId> <artifactId> <version> <packaging> <name> <description> <url> <licenses> <scm><url> <scm><connection> <developers> 6. Add Sonatype parent to your parent POM: <parent> <groupId>org.sonatype.oss</groupId> <artifactId>oss-parent</artifactId> <version>7</version> </parent> 7. Add SCM information to your project. For example like this: <scm> <connection>scm:git:git@github.com:sematext/ActionGenerator.git</connection> <developerConnection>scm:git:git@github.com:sematext/ActionGenerator.git</developerConnection> <url> https://github.com/sematext/ActionGenerator </url> </scm> 8. Modify your settings.xml file (you should have it in your maven home directory) and add the following: <servers> <server> <id>sonatype-nexus-snapshots</id> <username>your-jira-id</username> <password>your-jira-pwd</password> </server> <server> <id>sonatype-nexus-staging</id> <username>your-jira-id</username> <password>your-jira-pwd</password> </server> </servers> Where: your-jira-id is the ID of your account created in step 1 your-jira-pwd is the password for account created in step 1 9. Build your project, sign artifacts and deploy: $mvn clean install deploy -DperformRelease=true 10. After that, your artifcts should be uloaded to staging sonatype. You now should login into https://oss.sonatype.org/ with the previously created account, choose repository and click "Close". If you did everything right, close will be successful. 11. Now click "Release" on the closed repository and comment on the issue created in step 2) that you released the artifacts. Someone at Sonatype will review the artifacts and accept it. The review process will only happen once and the initial release. Later releases will be released automaticaly. This is what I did. Now I'm waiting for ActionGenerator to be released. In case you would like to go deeper into that process, there is a nice documentation available at https://docs.sonatype.org/display/Repository/Sonatype+OSS+Maven+Repository+Usage+Guide
          Hide
          Neha Narkhede added a comment -
          • How come this is not Fix Version 0.8? Should it not be? Is it not planned for the upcoming 0.8?
            Just changed the affected version to 0.8
          • Nobody is assigned. Is it known who will do this?
            Anyone who is interested can pick this up. Feel free to assign this to yourself.
          • It looks like KAFKA-134 is marked as blocker, but it's also not assigned and is not set for 0.8...?
            KAFKA-134 is required only to be able to publish to maven using an sbt command. For 0.8, if that is not ready, we can manually publish it. All the assignee would do in this case, is figure out the procedure to get this done and post it here. One of the committers can help roll the ball.
          Show
          Neha Narkhede added a comment - How come this is not Fix Version 0.8? Should it not be? Is it not planned for the upcoming 0.8? Just changed the affected version to 0.8 Nobody is assigned. Is it known who will do this? Anyone who is interested can pick this up. Feel free to assign this to yourself. It looks like KAFKA-134 is marked as blocker, but it's also not assigned and is not set for 0.8...? KAFKA-134 is required only to be able to publish to maven using an sbt command. For 0.8, if that is not ready, we can manually publish it. All the assignee would do in this case, is figure out the procedure to get this done and post it here. One of the committers can help roll the ball.
          Hide
          Otis Gospodnetic added a comment -

          FYI just emailed Stefan asking if zkclient 0.2 could be cut and published in Maven. He replied a few minutes later from a cab and CCed his colleague asking him to do this, so if stars align, you may see zkclient 0.2 out and you can remove it from Kafka.

          And then I really really hope you can publish Kafka to a Maven repo!

          Questions/comments:

          • How come this is not Fix Version 0.8? Should it not be? Is it not planned for the upcoming 0.8?
          • Nobody is assigned. Is it known who will do this?
          • It looks like KAFKA-134 is marked as blocker, but it's also not assigned and is not set for 0.8...?
          Show
          Otis Gospodnetic added a comment - FYI just emailed Stefan asking if zkclient 0.2 could be cut and published in Maven. He replied a few minutes later from a cab and CCed his colleague asking him to do this, so if stars align, you may see zkclient 0.2 out and you can remove it from Kafka. And then I really really hope you can publish Kafka to a Maven repo! Questions/comments: How come this is not Fix Version 0.8? Should it not be? Is it not planned for the upcoming 0.8? Nobody is assigned. Is it known who will do this? It looks like KAFKA-134 is marked as blocker, but it's also not assigned and is not set for 0.8...?
          Hide
          Neha Narkhede added a comment -

          >> So is the blocker for this issue the fact that there is a custom zkclient?
          I believe so.

          >> Should the modified API be provided as a patch to the original zkclient?
          It is already checked into zkclient trunk

          >> Should the zkclient maintainers be nudged to publish the new zkclient to a Maven repo?
          Sure, please feel free to ping them.

          Show
          Neha Narkhede added a comment - >> So is the blocker for this issue the fact that there is a custom zkclient? I believe so. >> Should the modified API be provided as a patch to the original zkclient? It is already checked into zkclient trunk >> Should the zkclient maintainers be nudged to publish the new zkclient to a Maven repo? Sure, please feel free to ping them.
          Hide
          Otis Gospodnetic added a comment -

          Thanks Neha.
          So is the blocker for this issue the fact that there is a custom zkclient?
          Should the modified API be provided as a patch to the original zkclient?
          Should the zkclient maintainers be nudged to publish the new zkclient to a Maven repo?

          Show
          Otis Gospodnetic added a comment - Thanks Neha. So is the blocker for this issue the fact that there is a custom zkclient? Should the modified API be provided as a patch to the original zkclient? Should the zkclient maintainers be nudged to publish the new zkclient to a Maven repo?
          Hide
          Neha Narkhede added a comment -

          >> Custom zkclient jar... maybe a Kafka developer in the know can jump in?

          That's because we modified one of the zk client APIs to expose the stat object returned from the underlying zookeeper APIs. But after that, zkclient was never published to Maven.

          Show
          Neha Narkhede added a comment - >> Custom zkclient jar... maybe a Kafka developer in the know can jump in? That's because we modified one of the zk client APIs to expose the stat object returned from the underlying zookeeper APIs. But after that, zkclient was never published to Maven.
          Hide
          Pierre-Yves Ritschard added a comment -

          the zkclient dep moved to maven a while back it is not modified IIRC

          On Tue, Nov 13, 2012 at 1:45 AM, Otis Gospodnetic (JIRA)

          Show
          Pierre-Yves Ritschard added a comment - the zkclient dep moved to maven a while back it is not modified IIRC On Tue, Nov 13, 2012 at 1:45 AM, Otis Gospodnetic (JIRA)
          Hide
          Otis Gospodnetic added a comment -

          I noticed Fix Version is not set, but if you look at https://issues.apache.org/jira/browse/KAFKA#selectedTab=com.atlassian.jira.plugin.system.project%3Apopularissues-panel you'll see this is THE most popular Kafka issue.

          Shouldn't this be set to 0.8, so it gets done for/with the upcoming release?

          Show
          Otis Gospodnetic added a comment - I noticed Fix Version is not set, but if you look at https://issues.apache.org/jira/browse/KAFKA#selectedTab=com.atlassian.jira.plugin.system.project%3Apopularissues-panel you'll see this is THE most popular Kafka issue. Shouldn't this be set to 0.8, so it gets done for/with the upcoming release?
          Hide
          Otis Gospodnetic added a comment - - edited

          Coda Hale metrics does seem to be in Maven repo already, see: http://mvnrepository.com/artifact/com.yammer.metrics
          Though from KAFKA-585 it looks like there may be a dependency on Metrics 3.*, which is not in any Maven repo yet.

          Custom zkclient jar... maybe a Kafka developer in the know can jump in?
          Maybe this jar could be included/hidden in one of the Kafka jars, so it is not treated as external dependencies?
          Oh, is that zkclient Stefan's zkclient? This one: http://mvnrepository.com/artifact/com.github.sgroschupf/zkclient/0.1 but locally modified, jarred, and stuck in svn?
          Maybe we could instead provide a patch for zkclient and get zkclient maintainer(s) to apply it? I knows Stefan personally and could bug him about it.... he is a busy CEO these days, but maybe somebody else on his team could get this done.

          Show
          Otis Gospodnetic added a comment - - edited Coda Hale metrics does seem to be in Maven repo already, see: http://mvnrepository.com/artifact/com.yammer.metrics Though from KAFKA-585 it looks like there may be a dependency on Metrics 3.*, which is not in any Maven repo yet. Custom zkclient jar... maybe a Kafka developer in the know can jump in? Maybe this jar could be included/hidden in one of the Kafka jars, so it is not treated as external dependencies? Oh, is that zkclient Stefan's zkclient? This one: http://mvnrepository.com/artifact/com.github.sgroschupf/zkclient/0.1 but locally modified, jarred, and stuck in svn? Maybe we could instead provide a patch for zkclient and get zkclient maintainer(s) to apply it? I knows Stefan personally and could bug him about it.... he is a busy CEO these days, but maybe somebody else on his team could get this done.
          Hide
          Sam Meder added a comment -

          The third party dependencies that I am aware of are:

          com.yammer.metrics, metrics-core
          com.yammer.metrics, metrics-annotation

          and a custom zkclient jar

          Show
          Sam Meder added a comment - The third party dependencies that I am aware of are: com.yammer.metrics, metrics-core com.yammer.metrics, metrics-annotation and a custom zkclient jar
          Hide
          Otis Gospodnetic added a comment -

          Maybe the thing to do then is to help owners of those projects Kafka depends on get their stuff into Maven, starting by opening tickets in their issue tracker.
          Sam Meder or Sam Meder - which dependencies are those specifically?

          Show
          Otis Gospodnetic added a comment - Maybe the thing to do then is to help owners of those projects Kafka depends on get their stuff into Maven, starting by opening tickets in their issue tracker. Sam Meder or Sam Meder - which dependencies are those specifically?
          Hide
          Sam Meder added a comment -

          One issue with doing this for the 0.8 branch is that it contains ~3 dependencies that are checked into the kafka source itself...

          Show
          Sam Meder added a comment - One issue with doing this for the 0.8 branch is that it contains ~3 dependencies that are checked into the kafka source itself...
          Hide
          Chris Aniszczyk added a comment -

          bump... +1

          Can this be done? We're looking to consume Kafka...

          At least to the Sonatype OSS repo or somewhere?
          https://docs.sonatype.org/display/Repository/Sonatype+OSS+Maven+Repository+Usage+Guide

          Show
          Chris Aniszczyk added a comment - bump... +1 Can this be done? We're looking to consume Kafka... At least to the Sonatype OSS repo or somewhere? https://docs.sonatype.org/display/Repository/Sonatype+OSS+Maven+Repository+Usage+Guide
          Hide
          Otis Gospodnetic added a comment -

          +1 for this. Funny, it looks like it has been exactly 365 days since the issue was opened and a bit over 6 months since the last comment. KAFKA-134 is still blocking... Jay Kreps, Joe Stein, & co, any chance of committing KAFKA-134 to unblock this issue (9 votes)?

          Show
          Otis Gospodnetic added a comment - +1 for this. Funny, it looks like it has been exactly 365 days since the issue was opened and a bit over 6 months since the last comment. KAFKA-134 is still blocking... Jay Kreps , Joe Stein , & co, any chance of committing KAFKA-134 to unblock this issue (9 votes)?
          Hide
          Thomas Dudziak added a comment -

          Fwiw, this is the pom I currently use for uploading to our internal maven repo.

          Show
          Thomas Dudziak added a comment - Fwiw, this is the pom I currently use for uploading to our internal maven repo.
          Hide
          Thomas Dudziak added a comment -

          Also, instead of having sbt (and pom) sections that exclude log4j 1.2.15's pulled-in dependency, you could simply update to log4j 1.2.16

          Show
          Thomas Dudziak added a comment - Also, instead of having sbt (and pom) sections that exclude log4j 1.2.15's pulled-in dependency, you could simply update to log4j 1.2.16
          Hide
          Pierre-Yves Ritschard added a comment -

          The pom.xml contained in this gist, coupled with the artifact built by the source contents of the kafka-0.7-incubating-src archive can be deployed to a maven repository and ease things for java clients.

          Show
          Pierre-Yves Ritschard added a comment - The pom.xml contained in this gist, coupled with the artifact built by the source contents of the kafka-0.7-incubating-src archive can be deployed to a maven repository and ease things for java clients.
          Hide
          Neha Narkhede added a comment -

          Patricio,

          This ticket is blocked on KAFKA-134. Once that is resolved, I believe this ticket will be easier to resolve.

          Thanks,
          Neha

          Show
          Neha Narkhede added a comment - Patricio, This ticket is blocked on KAFKA-134 . Once that is resolved, I believe this ticket will be easier to resolve. Thanks, Neha
          Hide
          Patricio Echague added a comment -

          Hi, has there been any progress on this ticket ?

          a pom file in place will be very useful to set up a project.

          Thanks for the effort here.

          Show
          Patricio Echague added a comment - Hi, has there been any progress on this ticket ? a pom file in place will be very useful to set up a project. Thanks for the effort here.
          Hide
          Joshua Hartman added a comment -

          I came into Mountain View today on a loooong train ride and my internet died right after I posted this. I wanted to say that it wasn't quite ready, but that we should sit down today and go through this so we can perfect it for kafka's long term needs.

          Chris Riccomini reminded me about this on Friday and I promised myself I would finish it over the weekend - I think I'm about 3 weeks overdue.

          Show
          Joshua Hartman added a comment - I came into Mountain View today on a loooong train ride and my internet died right after I posted this. I wanted to say that it wasn't quite ready, but that we should sit down today and go through this so we can perfect it for kafka's long term needs. Chris Riccomini reminded me about this on Friday and I promised myself I would finish it over the weekend - I think I'm about 3 weeks overdue.
          Hide
          Neha Narkhede added a comment -

          Thanks for the patch Josh ! Well in time for 0.7 release.

          Please could you attach the diff file here and also select the "grant it to Apache" option ? That way we can apply, test and accept the patch.

          Show
          Neha Narkhede added a comment - Thanks for the patch Josh ! Well in time for 0.7 release. Please could you attach the diff file here and also select the "grant it to Apache" option ? That way we can apply, test and accept the patch.
          Hide
          Jay Kreps added a comment -

          Josh, you are seriously my hero. Drinks are owed.

          Show
          Jay Kreps added a comment - Josh, you are seriously my hero. Drinks are owed.
          Hide
          Joshua Hartman added a comment -

          commit e6f067d538d4902f64f8397256fe8d967e036bc6
          Author: Joshua Hartman <jhartman@linkedin.com>
          Date: Mon Oct 24 10:56:12 2011 -0700

          Initial attempt at sbt upgrade.

          diff --git a/lib/sbt-launch.jar b/lib/sbt-launch.jar
          deleted file mode 100644
          index 67ee369..0000000
          Binary files a/lib/sbt-launch.jar and /dev/null differ
          diff --git a/project/Build.scala b/project/Build.scala
          new file mode 100644
          index 0000000..085809b
          — /dev/null
          +++ b/project/Build.scala
          @@ -0,0 +1,147 @@
          +import sbt._
          +import Keys._
          +import xml.NodeSeq
          +
          +object BuildSettings {
          + val exclusions = Seq("javax", "jmxri", "jmxtools", "mail", "jms")
          + val ivyExclude = <dependencies>{exclusions.map ( e => <exclude module=

          {e}

          />)}</dependencies>
          +
          + val buildSettings = Defaults.defaultSettings ++ Seq (
          + organization := "kafka",
          + version := "0.7",
          + crossScalaVersions := Seq("2.8.1", "2.9.1"),
          + ivyXML := ivyExclude
          + )
          +
          + def buildPomExtra(pom: NodeSeq, name: String, desc: String) = {
          + pom ++ Seq(
          + <name>
          +

          {name}

          + </name>,
          + <description>
          +

          {desc}

          + </description>,
          + <url>http://incubator.apache.org/kafka</url>,
          + <licenses>
          + <license>
          + <name>Apache</name>
          + <url>http://svn.apache.org/repos/asf/incubator/kafka/trunk/LICENSE</url>
          + <distribution>repo</distribution>
          + </license>
          + </licenses>,
          + <scm>
          + <url>http://svn.apache.org/repos/asf/incubator/kafka/trunk/</url>
          + <connection>scm:svn:http://svn.apache.org/repos/asf/incubator/kafka/trunk</connection>
          + <developerConnection>scm:svn:https://foo.googlecode.com/svn/trunk/</developerConnection>
          + <connection>scm:git:git://github.com/linkedin-sna/norbert.git</connection>
          + </scm>,
          + <developers>
          + <developer>
          + <id>jkreps</id>
          + <name>Jay Kreps</name>
          + <url>http://www.linkedin.com/in/jaykreps</url>
          + </developer>
          + <developer>
          + <id>junrao</id>
          + <name>Jun Rao</name>
          + <url>http://www.linkedin.com/in/junrao</url>
          + </developer>
          + <developer>
          + <id>nehanarkhede</id>
          + <name>Joshua Hartman</name>
          + <url>http://www.linkedin.com/in/nehanarkhede</url>
          + </developer>
          + </developers>
          + )
          + }
          +}
          +
          +object Resolvers

          { + val oracleRepo = "Oracle Maven 2 Repository" at "http://download.oracle.com/maven" + val jBossRepo = "JBoss Maven 2 Repository" at "http://repository.jboss.com/maven2" + val kafkaResolvers = Seq(oracleRepo, jBossRepo) +}

          +
          +object CoreDependencies

          { +// TODO jhartman: When sbt 0.11.1 is ready, we can use the following code instead of ivy xml +// val exclusions = Seq("javax", "jmxri", "jmxtools", "mail", "jms") map (n => ExclusionRule(name = n)) +// val log4j = ("log4j" % "log4j" % "1.2.15") excludeAll (exclusions :_*) + + val log4j = ("log4j" % "log4j" % "1.2.15") + val jopt = "net.sf.jopt-simple" % "jopt-simple" % "3.2" + val deps = Seq(log4j, jopt) +}

          +
          +object HadoopProducerDependencies

          { + val avro = "org.apache.avro" % "avro" % "1.4.1" + val jacksonCore = "org.codehaus.jackson" % "jackson-core-asl" % "1.5.5" + val jacksonMapper = "org.codehaus.jackson" % "jackson-mapper-asl" % "1.5.5" + val deps = Seq(avro, jacksonCore, jacksonMapper) +}

          +
          +object HadoopConsumerDependencies

          { + val jodaTime = "joda-time" % "joda-time" % "1.6" + val httpclient = "commons-httpclient" % "commons-httpclient" % "3.1" + val deps = Seq(jodaTime, httpclient) +}

          +
          +object TestDependencies

          { + val easymock = "org.easymock" % "easymock" % "3.0" % "test" + val junit = "junit" % "junit" % "4.1" % "test" + val scalaTest = "org.scalatest" % "scalatest" % "1.2" % "test" + val deps = Seq(easymock, junit, scalaTest) +}

          +
          +object KafkaBuild extends Build {
          + import BuildSettings._
          +
          + lazy val core = Project("core", file("core"),
          + settings = buildSettings ++ Seq(
          + libraryDependencies ++= CoreDependencies.deps ++ TestDependencies.deps,
          + resolvers := Resolvers.kafkaResolvers
          + )
          + )
          +
          + lazy val examples = Project("examples", file("examples"),
          + settings = buildSettings
          + ) dependsOn (core)
          +
          +
          + lazy val perf = Project("perf", file("perf"),
          + settings = buildSettings
          + ) dependsOn (core)
          +
          + lazy val hadoopProducer = Project("hadoop-producer", file("hadoop-producer"),
          + settings = buildSettings ++ Seq(
          + libraryDependencies ++= HadoopProducerDependencies.deps
          + )
          + ) dependsOn (core)
          +
          + lazy val hadoopConsumer = Project("hadoop-consumer", file("hadoop-consumer"),
          + settings = buildSettings ++ Seq(
          + libraryDependencies ++= HadoopConsumerDependencies.deps
          + )
          + ) dependsOn (core)
          +
          + lazy val contrib = Project("contrib", file("contrib"), settings = buildSettings) aggregate(hadoopConsumer, hadoopProducer)
          +
          + lazy val root = Project("root", file("."),
          + settings = buildSettings ++ Seq(
          + pomExtra <<= (pomExtra, name, description)

          { buildPomExtra }
          + )) aggregate(core, examples, perf, contrib)
          +
          + lazy val full = Project(
          + id = "kafka",
          + base = file("full"),
          + settings = buildSettings ++ Seq(
          + description := "Includes all of kafka project in one",
          + libraryDependencies ++= CoreDependencies.deps ++ TestDependencies.deps ++ HadoopProducerDependencies.deps ++ HadoopConsumerDependencies.deps,
          +
          + (unmanagedJars in Compile) <<= (projects.map(unmanagedJars in Compile in ).join).map(.flatten),
          + (unmanagedSourceDirectories in Compile) <<= projects.map(unmanagedSourceDirectories in Compile in ).join.apply(.flatten),
          + (managedSourceDirectories in Compile) <<= projects.map(managedSourceDirectories in Compile in ).join.apply(.flatten),
          +
          + pomExtra <<= (pomExtra, name, description) { buildPomExtra }

          + )
          + )
          +}
          diff --git a/project/build.properties b/project/build.properties
          deleted file mode 100644
          index 7c14cce..0000000
          — a/project/build.properties
          +++ /dev/null
          @@ -1,10 +0,0 @@
          -#Project properties
          -#Mon Feb 28 11:55:49 PST 2011
          -project.name=Kafka
          -sbt.version=0.7.5
          -project.version=0.7
          -build.scala.versions=2.8.0
          -contrib.root.dir=contrib
          -lib.dir=lib
          -target.dir=target/scala_2.8.0
          -dist.dir=dist
          diff --git a/project/build/KafkaProject.scala b/project/build/KafkaProject.scala
          index ba682d5..77aafba 100644
          — a/project/build/KafkaProject.scala
          +++ b/project/build/KafkaProject.scala
          @@ -1,190 +1,190 @@
          -import sbt._
          -
          -class KafkaProject(info: ProjectInfo) extends ParentProject(info) with IdeaProject {

          • lazy val core = project("core", "core-kafka", new CoreKafkaProject(_))
          • lazy val examples = project("examples", "java-examples", new KafkaExamplesProject(_), core)
          • lazy val perf = project("perf", "perf", new KafkaPerfProject(_), core)
          • lazy val contrib = project("contrib", "contrib", new ContribProject(_))
            -
          • lazy val releaseZipTask = core.packageDistTask
            -
          • val releaseZipDescription = "Compiles every sub project, runs unit tests, creates a deployable release zip file with dependencies, config, and scripts."
          • lazy val releaseZip = releaseZipTask dependsOn(core.corePackageAction, core.test, examples.examplesPackageAction, perf.perfPackageAction,
          • contrib.producerPackageAction, contrib.consumerPackageAction) describedAs releaseZipDescription
            -
          • class CoreKafkaProject(info: ProjectInfo) extends DefaultProject(info)
          • with IdeaProject with CoreDependencies with TestDependencies {
          • val corePackageAction = packageAllAction
            -
          • //The issue is going from log4j 1.2.14 to 1.2.15, the developers added some features which required
          • // some dependencies on various sun and javax packages.
          • override def ivyXML =
          • <dependencies>
          • <exclude module="javax"/>
          • <exclude module="jmxri"/>
          • <exclude module="jmxtools"/>
          • <exclude module="mail"/>
          • <exclude module="jms"/>
          • </dependencies>
            -
          • override def repositories = Set(ScalaToolsSnapshots, "JBoss Maven 2 Repository" at "http://repository.jboss.com/maven2",
          • "Oracle Maven 2 Repository" at "http://download.oracle.com/maven", "maven.org" at "http://repo2.maven.org/maven2/")
            -
          • override def artifactID = "kafka"
          • override def filterScalaJars = false
            -
          • // build the executable jar's classpath.
          • // (why is it necessary to explicitly remove the target/ {classes,resources} paths? hm.)
            - def dependentJars = {
            - val jars =
            - publicClasspath +++ mainDependencies.scalaJars — mainCompilePath — mainResourcesOutputPath
            - if (jars.get.find { jar => jar.name.startsWith("scala-library-") }.isDefined) {
            - // workaround bug in sbt: if the compiler is explicitly included, don't include 2 versions
            - // of the library.
            - jars — jars.filter { jar => - jar.absolutePath.contains("/boot/") && jar.name == "scala-library.jar" - }
            - } else { - jars - }
            - }
            -
            - def dependentJarNames = dependentJars.getFiles.map(.getName).filter(.endsWith(".jar"))
            - override def manifestClassPath = Some(dependentJarNames.map { "libs/" + _ }.mkString(" "))
            -
            - def distName = (artifactID + "-" + projectVersion.value)
            - def distPath = "dist" / distName ##
            -
            - def configPath = "config" ##
            - def configOutputPath = distPath / "config"
            -
            - def binPath = "bin" ##
            - def binOutputPath = distPath / "bin"
            -
            - def distZipName = { - "%s-%s.zip".format(artifactID, projectVersion.value) - }
            -
            - lazy val packageDistTask = task { - distPath.asFile.mkdirs() - (distPath / "libs").asFile.mkdirs() - binOutputPath.asFile.mkdirs() - configOutputPath.asFile.mkdirs() - - FileUtilities.copyFlat(List(jarPath), distPath, log).left.toOption orElse - FileUtilities.copyFlat(dependentJars.get, distPath / "libs", log).left.toOption orElse - FileUtilities.copy((configPath ***).get, configOutputPath, log).left.toOption orElse - FileUtilities.copy((binPath ***).get, binOutputPath, log).left.toOption orElse - FileUtilities.zip((("dist" / distName) ##).get, "dist" / distZipName, true, log) - None - }
            -
            - val PackageDistDescription = "Creates a deployable zip file with dependencies, config, and scripts."
            - lazy val packageDist = packageDistTask dependsOn(`package`, `test`) describedAs PackageDistDescription
            -
            - val cleanDist = cleanTask("dist" ##) describedAs("Erase any packaged distributions.")
            - override def cleanAction = super.cleanAction dependsOn(cleanDist)
            -
            - override def javaCompileOptions = super.javaCompileOptions ++
            - List(JavaCompileOption("-source"), JavaCompileOption("1.5"))
            - }
            -
            - class KafkaExamplesProject(info: ProjectInfo) extends DefaultProject(info)
            - with IdeaProject
            - with CoreDependencies { - val examplesPackageAction = packageAllAction - val dependsOnCore = core - //The issue is going from log4j 1.2.14 to 1.2.15, the developers added some features which required - // some dependencies on various sun and javax packages. - override def ivyXML = - <dependencies> - <exclude module="javax"/> - <exclude module="jmxri"/> - <exclude module="jmxtools"/> - <exclude module="mail"/> - <exclude module="jms"/> - </dependencies> - - override def artifactID = "kafka-java-examples" - override def filterScalaJars = false - }
            -
            - class KafkaPerfProject(info: ProjectInfo) extends DefaultProject(info)
            - with IdeaProject
            - with CoreDependencies { - val perfPackageAction = packageAllAction - val dependsOnCore = core - //The issue is going from log4j 1.2.14 to 1.2.15, the developers added some features which required - // some dependencies on various sun and javax packages. - override def ivyXML = - <dependencies> - <exclude module="javax"/> - <exclude module="jmxri"/> - <exclude module="jmxtools"/> - <exclude module="mail"/> - <exclude module="jms"/> - </dependencies> - - override def artifactID = "kafka-perf" - override def filterScalaJars = false - }
            -
            - class ContribProject(info: ProjectInfo) extends ParentProject(info) with IdeaProject {
            - lazy val hadoopProducer = project("hadoop-producer", "hadoop producer",
            - new HadoopProducerProject(_), core)
            - lazy val hadoopConsumer = project("hadoop-consumer", "hadoop consumer",
            - new HadoopConsumerProject(_), core)
            -
            - val producerPackageAction = hadoopProducer.producerPackageAction
            - val consumerPackageAction = hadoopConsumer.consumerPackageAction
            -
            - class HadoopProducerProject(info: ProjectInfo) extends DefaultProject(info)
            - with IdeaProject
            - with CoreDependencies { - val producerPackageAction = packageAllAction - override def ivyXML = - <dependencies> - <exclude module="netty"/> - <exclude module="javax"/> - <exclude module="jmxri"/> - <exclude module="jmxtools"/> - <exclude module="mail"/> - <exclude module="jms"/> - </dependencies> - - val avro = "org.apache.avro" % "avro" % "1.4.1" - val jacksonCore = "org.codehaus.jackson" % "jackson-core-asl" % "1.5.5" - val jacksonMapper = "org.codehaus.jackson" % "jackson-mapper-asl" % "1.5.5" - }
            -
            - class HadoopConsumerProject(info: ProjectInfo) extends DefaultProject(info)
            - with IdeaProject
            - with CoreDependencies { - val consumerPackageAction = packageAllAction - override def ivyXML = - <dependencies> - <exclude module="netty"/> - <exclude module="javax"/> - <exclude module="jmxri"/> - <exclude module="jmxtools"/> - <exclude module="mail"/> - <exclude module="jms"/> - </dependencies> - - val jodaTime = "joda-time" % "joda-time" % "1.6" - val httpclient = "commons-httpclient" % "commons-httpclient" % "3.1" - }
            - }
            -
            - trait TestDependencies { - val easymock = "org.easymock" % "easymock" % "3.0" % "test" - val junit = "junit" % "junit" % "4.1" % "test" - val scalaTest = "org.scalatest" % "scalatest" % "1.2" % "test" - }
            -
            - trait CoreDependencies { - val log4j = "log4j" % "log4j" % "1.2.15" - val jopt = "net.sf.jopt-simple" % "jopt-simple" % "3.2" - }
            -
            -}
            +//import sbt._
            +//
            +//class KafkaProject(info: ProjectInfo) extends ParentProject(info) with IdeaProject {
            +// lazy val core = project("core", "core-kafka", new CoreKafkaProject(_))
            +// lazy val examples = project("examples", "java-examples", new KafkaExamplesProject(_), core)
            +// lazy val perf = project("perf", "perf", new KafkaPerfProject(_), core)
            +// lazy val contrib = project("contrib", "contrib", new ContribProject(_))
            +//
            +// lazy val releaseZipTask = core.packageDistTask
            +//
            +// val releaseZipDescription = "Compiles every sub project, runs unit tests, creates a deployable release zip file with dependencies, config, and scripts."
            +// lazy val releaseZip = releaseZipTask dependsOn(core.corePackageAction, core.test, examples.examplesPackageAction, perf.perfPackageAction,
            +// contrib.producerPackageAction, contrib.consumerPackageAction) describedAs releaseZipDescription
            +//
            +// class CoreKafkaProject(info: ProjectInfo) extends DefaultProject(info)
            +// with IdeaProject with CoreDependencies with TestDependencies {
            +// val corePackageAction = packageAllAction
            +//
            +// //The issue is going from log4j 1.2.14 to 1.2.15, the developers added some features which required
            +// // some dependencies on various sun and javax packages.
            +// override def ivyXML =
            +// <dependencies>
            +// <exclude module="javax"/>
            +// <exclude module="jmxri"/>
            +// <exclude module="jmxtools"/>
            +// <exclude module="mail"/>
            +// <exclude module="jms"/>
            +// </dependencies>
            +//
            +// override def repositories = Set(ScalaToolsSnapshots, "JBoss Maven 2 Repository" at "http://repository.jboss.com/maven2",
            +// "Oracle Maven 2 Repository" at "http://download.oracle.com/maven", "maven.org" at "http://repo2.maven.org/maven2/")
            +//
            +// override def artifactID = "kafka"
            +// override def filterScalaJars = false
            +//
            +// // build the executable jar's classpath.
            +// // (why is it necessary to explicitly remove the target/{classes,resources}

            paths? hm.)
            +// def dependentJars = {
            +// val jars =
            +// publicClasspath +++ mainDependencies.scalaJars — mainCompilePath — mainResourcesOutputPath
            +// if (jars.get.find

            { jar => jar.name.startsWith("scala-library-") }

            .isDefined)

            Unknown macro: {+// // workaround bug in sbt}

            else

            { +// jars +// }

            +// }
            +//
            +// def dependentJarNames = dependentJars.getFiles.map(.getName).filter(.endsWith(".jar"))
            +// override def manifestClassPath = Some(dependentJarNames.map

            { "libs/" + _ }

            .mkString(" "))
            +//
            +// def distName = (artifactID + "-" + projectVersion.value)
            +// def distPath = "dist" / distName ##
            +//
            +// def configPath = "config" ##
            +// def configOutputPath = distPath / "config"
            +//
            +// def binPath = "bin" ##
            +// def binOutputPath = distPath / "bin"
            +//
            +// def distZipName =

            { +// "%s-%s.zip".format(artifactID, projectVersion.value) +// }

            +//
            +// lazy val packageDistTask = task

            { +// distPath.asFile.mkdirs() +// (distPath / "libs").asFile.mkdirs() +// binOutputPath.asFile.mkdirs() +// configOutputPath.asFile.mkdirs() +// +// FileUtilities.copyFlat(List(jarPath), distPath, log).left.toOption orElse +// FileUtilities.copyFlat(dependentJars.get, distPath / "libs", log).left.toOption orElse +// FileUtilities.copy((configPath ***).get, configOutputPath, log).left.toOption orElse +// FileUtilities.copy((binPath ***).get, binOutputPath, log).left.toOption orElse +// FileUtilities.zip((("dist" / distName) ##).get, "dist" / distZipName, true, log) +// None +// }

            +//
            +// val PackageDistDescription = "Creates a deployable zip file with dependencies, config, and scripts."
            +// lazy val packageDist = packageDistTask dependsOn(`package`, `test`) describedAs PackageDistDescription
            +//
            +// val cleanDist = cleanTask("dist" ##) describedAs("Erase any packaged distributions.")
            +// override def cleanAction = super.cleanAction dependsOn(cleanDist)
            +//
            +// override def javaCompileOptions = super.javaCompileOptions ++
            +// List(JavaCompileOption("-source"), JavaCompileOption("1.5"))
            +// }
            +//
            +// class KafkaExamplesProject(info: ProjectInfo) extends DefaultProject(info)
            +// with IdeaProject
            +// with CoreDependencies

            { +// val examplesPackageAction = packageAllAction +// val dependsOnCore = core +// //The issue is going from log4j 1.2.14 to 1.2.15, the developers added some features which required +// // some dependencies on various sun and javax packages. +// override def ivyXML = +// <dependencies> +// <exclude module="javax"/> +// <exclude module="jmxri"/> +// <exclude module="jmxtools"/> +// <exclude module="mail"/> +// <exclude module="jms"/> +// </dependencies> +// +// override def artifactID = "kafka-java-examples" +// override def filterScalaJars = false +// }

            +//
            +// class KafkaPerfProject(info: ProjectInfo) extends DefaultProject(info)
            +// with IdeaProject
            +// with CoreDependencies

            { +// val perfPackageAction = packageAllAction +// val dependsOnCore = core +// //The issue is going from log4j 1.2.14 to 1.2.15, the developers added some features which required +// // some dependencies on various sun and javax packages. +// override def ivyXML = +// <dependencies> +// <exclude module="javax"/> +// <exclude module="jmxri"/> +// <exclude module="jmxtools"/> +// <exclude module="mail"/> +// <exclude module="jms"/> +// </dependencies> +// +// override def artifactID = "kafka-perf" +// override def filterScalaJars = false +// }

            +//
            +// class ContribProject(info: ProjectInfo) extends ParentProject(info) with IdeaProject

            Unknown macro: {+// lazy val hadoopProducer = project("hadoop-producer", "hadoop producer",+// new HadoopProducerProject(_), core)+// lazy val hadoopConsumer = project("hadoop-consumer", "hadoop consumer",+// new HadoopConsumerProject(_), core)+//+// val producerPackageAction = hadoopProducer.producerPackageAction+// val consumerPackageAction = hadoopConsumer.consumerPackageAction+//+// class HadoopProducerProject(info}

            +//
            +// trait TestDependencies

            { +// val easymock = "org.easymock" % "easymock" % "3.0" % "test" +// val junit = "junit" % "junit" % "4.1" % "test" +// val scalaTest = "org.scalatest" % "scalatest" % "1.2" % "test" +// }

            +//
            +// trait CoreDependencies

            { +// val log4j = "log4j" % "log4j" % "1.2.15" +// val jopt = "net.sf.jopt-simple" % "jopt-simple" % "3.2" +// }

            +//
            +//}
            diff --git a/project/plugins/Plugins.scala b/project/plugins/Plugins.scala
            deleted file mode 100644
            index 6938e30..0000000

              • a/project/plugins/Plugins.scala
                +++ /dev/null
                @@ -1,6 +0,0 @@
                -import sbt._
                -
                -class Plugins(info: ProjectInfo) extends PluginDefinition(info) { - val repo = "GH-pages repo" at "http://mpeltonen.github.com/maven/" - val idea = "com.github.mpeltonen" % "sbt-idea-plugin" % "0.1-SNAPSHOT" -}

                diff --git a/sbt b/sbt
                index 7d8b41e..657e990 100755

              • a/sbt
                +++ b/sbt
                @@ -1 +1,2 @@
                -java -Xmx1024M -XX:MaxPermSize=512m -jar `dirname $0`/lib/sbt-launch.jar "$@"
                +java -Xmx1024M -XX:MaxPermSize=512m -jar `dirname $0`/sbt-launch.jar "$@"
                +
          Show
          Joshua Hartman added a comment - commit e6f067d538d4902f64f8397256fe8d967e036bc6 Author: Joshua Hartman <jhartman@linkedin.com> Date: Mon Oct 24 10:56:12 2011 -0700 Initial attempt at sbt upgrade. diff --git a/lib/sbt-launch.jar b/lib/sbt-launch.jar deleted file mode 100644 index 67ee369..0000000 Binary files a/lib/sbt-launch.jar and /dev/null differ diff --git a/project/Build.scala b/project/Build.scala new file mode 100644 index 0000000..085809b — /dev/null +++ b/project/Build.scala @@ -0,0 +1,147 @@ +import sbt._ +import Keys._ +import xml.NodeSeq + +object BuildSettings { + val exclusions = Seq("javax", "jmxri", "jmxtools", "mail", "jms") + val ivyExclude = <dependencies>{exclusions.map ( e => <exclude module= {e} />)}</dependencies> + + val buildSettings = Defaults.defaultSettings ++ Seq ( + organization := "kafka", + version := "0.7", + crossScalaVersions := Seq("2.8.1", "2.9.1"), + ivyXML := ivyExclude + ) + + def buildPomExtra(pom: NodeSeq, name: String, desc: String) = { + pom ++ Seq( + <name> + {name} + </name>, + <description> + {desc} + </description>, + <url> http://incubator.apache.org/kafka </url>, + <licenses> + <license> + <name>Apache</name> + <url> http://svn.apache.org/repos/asf/incubator/kafka/trunk/LICENSE </url> + <distribution>repo</distribution> + </license> + </licenses>, + <scm> + <url> http://svn.apache.org/repos/asf/incubator/kafka/trunk/ </url> + <connection>scm:svn: http://svn.apache.org/repos/asf/incubator/kafka/trunk </connection> + <developerConnection>scm:svn: https://foo.googlecode.com/svn/trunk/ </developerConnection> + <connection>scm:git:git://github.com/linkedin-sna/norbert.git</connection> + </scm>, + <developers> + <developer> + <id>jkreps</id> + <name>Jay Kreps</name> + <url> http://www.linkedin.com/in/jaykreps </url> + </developer> + <developer> + <id>junrao</id> + <name>Jun Rao</name> + <url> http://www.linkedin.com/in/junrao </url> + </developer> + <developer> + <id>nehanarkhede</id> + <name>Joshua Hartman</name> + <url> http://www.linkedin.com/in/nehanarkhede </url> + </developer> + </developers> + ) + } +} + +object Resolvers { + val oracleRepo = "Oracle Maven 2 Repository" at "http://download.oracle.com/maven" + val jBossRepo = "JBoss Maven 2 Repository" at "http://repository.jboss.com/maven2" + val kafkaResolvers = Seq(oracleRepo, jBossRepo) +} + +object CoreDependencies { +// TODO jhartman: When sbt 0.11.1 is ready, we can use the following code instead of ivy xml +// val exclusions = Seq("javax", "jmxri", "jmxtools", "mail", "jms") map (n => ExclusionRule(name = n)) +// val log4j = ("log4j" % "log4j" % "1.2.15") excludeAll (exclusions :_*) + + val log4j = ("log4j" % "log4j" % "1.2.15") + val jopt = "net.sf.jopt-simple" % "jopt-simple" % "3.2" + val deps = Seq(log4j, jopt) +} + +object HadoopProducerDependencies { + val avro = "org.apache.avro" % "avro" % "1.4.1" + val jacksonCore = "org.codehaus.jackson" % "jackson-core-asl" % "1.5.5" + val jacksonMapper = "org.codehaus.jackson" % "jackson-mapper-asl" % "1.5.5" + val deps = Seq(avro, jacksonCore, jacksonMapper) +} + +object HadoopConsumerDependencies { + val jodaTime = "joda-time" % "joda-time" % "1.6" + val httpclient = "commons-httpclient" % "commons-httpclient" % "3.1" + val deps = Seq(jodaTime, httpclient) +} + +object TestDependencies { + val easymock = "org.easymock" % "easymock" % "3.0" % "test" + val junit = "junit" % "junit" % "4.1" % "test" + val scalaTest = "org.scalatest" % "scalatest" % "1.2" % "test" + val deps = Seq(easymock, junit, scalaTest) +} + +object KafkaBuild extends Build { + import BuildSettings._ + + lazy val core = Project("core", file("core"), + settings = buildSettings ++ Seq( + libraryDependencies ++= CoreDependencies.deps ++ TestDependencies.deps, + resolvers := Resolvers.kafkaResolvers + ) + ) + + lazy val examples = Project("examples", file("examples"), + settings = buildSettings + ) dependsOn (core) + + + lazy val perf = Project("perf", file("perf"), + settings = buildSettings + ) dependsOn (core) + + lazy val hadoopProducer = Project("hadoop-producer", file("hadoop-producer"), + settings = buildSettings ++ Seq( + libraryDependencies ++= HadoopProducerDependencies.deps + ) + ) dependsOn (core) + + lazy val hadoopConsumer = Project("hadoop-consumer", file("hadoop-consumer"), + settings = buildSettings ++ Seq( + libraryDependencies ++= HadoopConsumerDependencies.deps + ) + ) dependsOn (core) + + lazy val contrib = Project("contrib", file("contrib"), settings = buildSettings) aggregate(hadoopConsumer, hadoopProducer) + + lazy val root = Project("root", file("."), + settings = buildSettings ++ Seq( + pomExtra <<= (pomExtra, name, description) { buildPomExtra } + )) aggregate(core, examples, perf, contrib) + + lazy val full = Project( + id = "kafka", + base = file("full"), + settings = buildSettings ++ Seq( + description := "Includes all of kafka project in one", + libraryDependencies ++= CoreDependencies.deps ++ TestDependencies.deps ++ HadoopProducerDependencies.deps ++ HadoopConsumerDependencies.deps, + + (unmanagedJars in Compile) <<= (projects.map(unmanagedJars in Compile in ).join).map( .flatten), + (unmanagedSourceDirectories in Compile) <<= projects.map(unmanagedSourceDirectories in Compile in ).join.apply( .flatten), + (managedSourceDirectories in Compile) <<= projects.map(managedSourceDirectories in Compile in ).join.apply( .flatten), + + pomExtra <<= (pomExtra, name, description) { buildPomExtra } + ) + ) +} diff --git a/project/build.properties b/project/build.properties deleted file mode 100644 index 7c14cce..0000000 — a/project/build.properties +++ /dev/null @@ -1,10 +0,0 @@ -#Project properties -#Mon Feb 28 11:55:49 PST 2011 -project.name=Kafka -sbt.version=0.7.5 -project.version=0.7 -build.scala.versions=2.8.0 -contrib.root.dir=contrib -lib.dir=lib -target.dir=target/scala_2.8.0 -dist.dir=dist diff --git a/project/build/KafkaProject.scala b/project/build/KafkaProject.scala index ba682d5..77aafba 100644 — a/project/build/KafkaProject.scala +++ b/project/build/KafkaProject.scala @@ -1,190 +1,190 @@ -import sbt._ - -class KafkaProject(info: ProjectInfo) extends ParentProject(info) with IdeaProject { lazy val core = project("core", "core-kafka", new CoreKafkaProject(_)) lazy val examples = project("examples", "java-examples", new KafkaExamplesProject(_), core) lazy val perf = project("perf", "perf", new KafkaPerfProject(_), core) lazy val contrib = project("contrib", "contrib", new ContribProject(_)) - lazy val releaseZipTask = core.packageDistTask - val releaseZipDescription = "Compiles every sub project, runs unit tests, creates a deployable release zip file with dependencies, config, and scripts." lazy val releaseZip = releaseZipTask dependsOn(core.corePackageAction, core.test, examples.examplesPackageAction, perf.perfPackageAction, contrib.producerPackageAction, contrib.consumerPackageAction) describedAs releaseZipDescription - class CoreKafkaProject(info: ProjectInfo) extends DefaultProject(info) with IdeaProject with CoreDependencies with TestDependencies { val corePackageAction = packageAllAction - //The issue is going from log4j 1.2.14 to 1.2.15, the developers added some features which required // some dependencies on various sun and javax packages. override def ivyXML = <dependencies> <exclude module="javax"/> <exclude module="jmxri"/> <exclude module="jmxtools"/> <exclude module="mail"/> <exclude module="jms"/> </dependencies> - override def repositories = Set(ScalaToolsSnapshots, "JBoss Maven 2 Repository" at "http://repository.jboss.com/maven2", "Oracle Maven 2 Repository" at "http://download.oracle.com/maven", "maven.org" at "http://repo2.maven.org/maven2/") - override def artifactID = "kafka" override def filterScalaJars = false - // build the executable jar's classpath. // (why is it necessary to explicitly remove the target/ {classes,resources} paths? hm.) - def dependentJars = { - val jars = - publicClasspath +++ mainDependencies.scalaJars — mainCompilePath — mainResourcesOutputPath - if (jars.get.find { jar => jar.name.startsWith("scala-library-") }.isDefined) { - // workaround bug in sbt: if the compiler is explicitly included, don't include 2 versions - // of the library. - jars — jars.filter { jar => - jar.absolutePath.contains("/boot/") && jar.name == "scala-library.jar" - } - } else { - jars - } - } - - def dependentJarNames = dependentJars.getFiles.map( .getName).filter( .endsWith(".jar")) - override def manifestClassPath = Some(dependentJarNames.map { "libs/" + _ }.mkString(" ")) - - def distName = (artifactID + "-" + projectVersion.value) - def distPath = "dist" / distName ## - - def configPath = "config" ## - def configOutputPath = distPath / "config" - - def binPath = "bin" ## - def binOutputPath = distPath / "bin" - - def distZipName = { - "%s-%s.zip".format(artifactID, projectVersion.value) - } - - lazy val packageDistTask = task { - distPath.asFile.mkdirs() - (distPath / "libs").asFile.mkdirs() - binOutputPath.asFile.mkdirs() - configOutputPath.asFile.mkdirs() - - FileUtilities.copyFlat(List(jarPath), distPath, log).left.toOption orElse - FileUtilities.copyFlat(dependentJars.get, distPath / "libs", log).left.toOption orElse - FileUtilities.copy((configPath ***).get, configOutputPath, log).left.toOption orElse - FileUtilities.copy((binPath ***).get, binOutputPath, log).left.toOption orElse - FileUtilities.zip((("dist" / distName) ##).get, "dist" / distZipName, true, log) - None - } - - val PackageDistDescription = "Creates a deployable zip file with dependencies, config, and scripts." - lazy val packageDist = packageDistTask dependsOn(`package`, `test`) describedAs PackageDistDescription - - val cleanDist = cleanTask("dist" ##) describedAs("Erase any packaged distributions.") - override def cleanAction = super.cleanAction dependsOn(cleanDist) - - override def javaCompileOptions = super.javaCompileOptions ++ - List(JavaCompileOption("-source"), JavaCompileOption("1.5")) - } - - class KafkaExamplesProject(info: ProjectInfo) extends DefaultProject(info) - with IdeaProject - with CoreDependencies { - val examplesPackageAction = packageAllAction - val dependsOnCore = core - //The issue is going from log4j 1.2.14 to 1.2.15, the developers added some features which required - // some dependencies on various sun and javax packages. - override def ivyXML = - <dependencies> - <exclude module="javax"/> - <exclude module="jmxri"/> - <exclude module="jmxtools"/> - <exclude module="mail"/> - <exclude module="jms"/> - </dependencies> - - override def artifactID = "kafka-java-examples" - override def filterScalaJars = false - } - - class KafkaPerfProject(info: ProjectInfo) extends DefaultProject(info) - with IdeaProject - with CoreDependencies { - val perfPackageAction = packageAllAction - val dependsOnCore = core - //The issue is going from log4j 1.2.14 to 1.2.15, the developers added some features which required - // some dependencies on various sun and javax packages. - override def ivyXML = - <dependencies> - <exclude module="javax"/> - <exclude module="jmxri"/> - <exclude module="jmxtools"/> - <exclude module="mail"/> - <exclude module="jms"/> - </dependencies> - - override def artifactID = "kafka-perf" - override def filterScalaJars = false - } - - class ContribProject(info: ProjectInfo) extends ParentProject(info) with IdeaProject { - lazy val hadoopProducer = project("hadoop-producer", "hadoop producer", - new HadoopProducerProject(_), core) - lazy val hadoopConsumer = project("hadoop-consumer", "hadoop consumer", - new HadoopConsumerProject(_), core) - - val producerPackageAction = hadoopProducer.producerPackageAction - val consumerPackageAction = hadoopConsumer.consumerPackageAction - - class HadoopProducerProject(info: ProjectInfo) extends DefaultProject(info) - with IdeaProject - with CoreDependencies { - val producerPackageAction = packageAllAction - override def ivyXML = - <dependencies> - <exclude module="netty"/> - <exclude module="javax"/> - <exclude module="jmxri"/> - <exclude module="jmxtools"/> - <exclude module="mail"/> - <exclude module="jms"/> - </dependencies> - - val avro = "org.apache.avro" % "avro" % "1.4.1" - val jacksonCore = "org.codehaus.jackson" % "jackson-core-asl" % "1.5.5" - val jacksonMapper = "org.codehaus.jackson" % "jackson-mapper-asl" % "1.5.5" - } - - class HadoopConsumerProject(info: ProjectInfo) extends DefaultProject(info) - with IdeaProject - with CoreDependencies { - val consumerPackageAction = packageAllAction - override def ivyXML = - <dependencies> - <exclude module="netty"/> - <exclude module="javax"/> - <exclude module="jmxri"/> - <exclude module="jmxtools"/> - <exclude module="mail"/> - <exclude module="jms"/> - </dependencies> - - val jodaTime = "joda-time" % "joda-time" % "1.6" - val httpclient = "commons-httpclient" % "commons-httpclient" % "3.1" - } - } - - trait TestDependencies { - val easymock = "org.easymock" % "easymock" % "3.0" % "test" - val junit = "junit" % "junit" % "4.1" % "test" - val scalaTest = "org.scalatest" % "scalatest" % "1.2" % "test" - } - - trait CoreDependencies { - val log4j = "log4j" % "log4j" % "1.2.15" - val jopt = "net.sf.jopt-simple" % "jopt-simple" % "3.2" - } - -} +//import sbt._ +// +//class KafkaProject(info: ProjectInfo) extends ParentProject(info) with IdeaProject { +// lazy val core = project("core", "core-kafka", new CoreKafkaProject(_)) +// lazy val examples = project("examples", "java-examples", new KafkaExamplesProject(_), core) +// lazy val perf = project("perf", "perf", new KafkaPerfProject(_), core) +// lazy val contrib = project("contrib", "contrib", new ContribProject(_)) +// +// lazy val releaseZipTask = core.packageDistTask +// +// val releaseZipDescription = "Compiles every sub project, runs unit tests, creates a deployable release zip file with dependencies, config, and scripts." +// lazy val releaseZip = releaseZipTask dependsOn(core.corePackageAction, core.test, examples.examplesPackageAction, perf.perfPackageAction, +// contrib.producerPackageAction, contrib.consumerPackageAction) describedAs releaseZipDescription +// +// class CoreKafkaProject(info: ProjectInfo) extends DefaultProject(info) +// with IdeaProject with CoreDependencies with TestDependencies { +// val corePackageAction = packageAllAction +// +// //The issue is going from log4j 1.2.14 to 1.2.15, the developers added some features which required +// // some dependencies on various sun and javax packages. +// override def ivyXML = +// <dependencies> +// <exclude module="javax"/> +// <exclude module="jmxri"/> +// <exclude module="jmxtools"/> +// <exclude module="mail"/> +// <exclude module="jms"/> +// </dependencies> +// +// override def repositories = Set(ScalaToolsSnapshots, "JBoss Maven 2 Repository" at "http://repository.jboss.com/maven2", +// "Oracle Maven 2 Repository" at "http://download.oracle.com/maven", "maven.org" at "http://repo2.maven.org/maven2/") +// +// override def artifactID = "kafka" +// override def filterScalaJars = false +// +// // build the executable jar's classpath. +// // (why is it necessary to explicitly remove the target/{classes,resources} paths? hm.) +// def dependentJars = { +// val jars = +// publicClasspath +++ mainDependencies.scalaJars — mainCompilePath — mainResourcesOutputPath +// if (jars.get.find { jar => jar.name.startsWith("scala-library-") } .isDefined) Unknown macro: {+// // workaround bug in sbt} else { +// jars +// } +// } +// +// def dependentJarNames = dependentJars.getFiles.map( .getName).filter( .endsWith(".jar")) +// override def manifestClassPath = Some(dependentJarNames.map { "libs/" + _ } .mkString(" ")) +// +// def distName = (artifactID + "-" + projectVersion.value) +// def distPath = "dist" / distName ## +// +// def configPath = "config" ## +// def configOutputPath = distPath / "config" +// +// def binPath = "bin" ## +// def binOutputPath = distPath / "bin" +// +// def distZipName = { +// "%s-%s.zip".format(artifactID, projectVersion.value) +// } +// +// lazy val packageDistTask = task { +// distPath.asFile.mkdirs() +// (distPath / "libs").asFile.mkdirs() +// binOutputPath.asFile.mkdirs() +// configOutputPath.asFile.mkdirs() +// +// FileUtilities.copyFlat(List(jarPath), distPath, log).left.toOption orElse +// FileUtilities.copyFlat(dependentJars.get, distPath / "libs", log).left.toOption orElse +// FileUtilities.copy((configPath ***).get, configOutputPath, log).left.toOption orElse +// FileUtilities.copy((binPath ***).get, binOutputPath, log).left.toOption orElse +// FileUtilities.zip((("dist" / distName) ##).get, "dist" / distZipName, true, log) +// None +// } +// +// val PackageDistDescription = "Creates a deployable zip file with dependencies, config, and scripts." +// lazy val packageDist = packageDistTask dependsOn(`package`, `test`) describedAs PackageDistDescription +// +// val cleanDist = cleanTask("dist" ##) describedAs("Erase any packaged distributions.") +// override def cleanAction = super.cleanAction dependsOn(cleanDist) +// +// override def javaCompileOptions = super.javaCompileOptions ++ +// List(JavaCompileOption("-source"), JavaCompileOption("1.5")) +// } +// +// class KafkaExamplesProject(info: ProjectInfo) extends DefaultProject(info) +// with IdeaProject +// with CoreDependencies { +// val examplesPackageAction = packageAllAction +// val dependsOnCore = core +// //The issue is going from log4j 1.2.14 to 1.2.15, the developers added some features which required +// // some dependencies on various sun and javax packages. +// override def ivyXML = +// <dependencies> +// <exclude module="javax"/> +// <exclude module="jmxri"/> +// <exclude module="jmxtools"/> +// <exclude module="mail"/> +// <exclude module="jms"/> +// </dependencies> +// +// override def artifactID = "kafka-java-examples" +// override def filterScalaJars = false +// } +// +// class KafkaPerfProject(info: ProjectInfo) extends DefaultProject(info) +// with IdeaProject +// with CoreDependencies { +// val perfPackageAction = packageAllAction +// val dependsOnCore = core +// //The issue is going from log4j 1.2.14 to 1.2.15, the developers added some features which required +// // some dependencies on various sun and javax packages. +// override def ivyXML = +// <dependencies> +// <exclude module="javax"/> +// <exclude module="jmxri"/> +// <exclude module="jmxtools"/> +// <exclude module="mail"/> +// <exclude module="jms"/> +// </dependencies> +// +// override def artifactID = "kafka-perf" +// override def filterScalaJars = false +// } +// +// class ContribProject(info: ProjectInfo) extends ParentProject(info) with IdeaProject Unknown macro: {+// lazy val hadoopProducer = project("hadoop-producer", "hadoop producer",+// new HadoopProducerProject(_), core)+// lazy val hadoopConsumer = project("hadoop-consumer", "hadoop consumer",+// new HadoopConsumerProject(_), core)+//+// val producerPackageAction = hadoopProducer.producerPackageAction+// val consumerPackageAction = hadoopConsumer.consumerPackageAction+//+// class HadoopProducerProject(info} +// +// trait TestDependencies { +// val easymock = "org.easymock" % "easymock" % "3.0" % "test" +// val junit = "junit" % "junit" % "4.1" % "test" +// val scalaTest = "org.scalatest" % "scalatest" % "1.2" % "test" +// } +// +// trait CoreDependencies { +// val log4j = "log4j" % "log4j" % "1.2.15" +// val jopt = "net.sf.jopt-simple" % "jopt-simple" % "3.2" +// } +// +//} diff --git a/project/plugins/Plugins.scala b/project/plugins/Plugins.scala deleted file mode 100644 index 6938e30..0000000 a/project/plugins/Plugins.scala +++ /dev/null @@ -1,6 +0,0 @@ -import sbt._ - -class Plugins(info: ProjectInfo) extends PluginDefinition(info) { - val repo = "GH-pages repo" at "http://mpeltonen.github.com/maven/" - val idea = "com.github.mpeltonen" % "sbt-idea-plugin" % "0.1-SNAPSHOT" -} diff --git a/sbt b/sbt index 7d8b41e..657e990 100755 a/sbt +++ b/sbt @@ -1 +1,2 @@ -java -Xmx1024M -XX:MaxPermSize=512m -jar `dirname $0`/lib/sbt-launch.jar "$@" +java -Xmx1024M -XX:MaxPermSize=512m -jar `dirname $0`/sbt-launch.jar "$@" +
          Hide
          Chris Riccomini added a comment -

          Hey Guys,

          Is there a maven repo for Kafka at this point?

          I sure could use it if there is one.

          Cheers,
          Chris

          Show
          Chris Riccomini added a comment - Hey Guys, Is there a maven repo for Kafka at this point? I sure could use it if there is one. Cheers, Chris
          Hide
          Pierre-Yves Ritschard added a comment -

          I think it would also be nice to publish the resulting clojure client jar to a maven repo, so client clojure code could benefit easily.

          Show
          Pierre-Yves Ritschard added a comment - I think it would also be nice to publish the resulting clojure client jar to a maven repo, so client clojure code could benefit easily.
          Hide
          Jay Kreps added a comment -

          Josh, you are my hero.

          Show
          Jay Kreps added a comment - Josh, you are my hero.
          Hide
          Alan Cabrera added a comment - - edited

          I need to state the obvious, because I am a mentor, but while it's good to get this machinery working, this project cannot actually publish any artifacts until its first official release.

          Show
          Alan Cabrera added a comment - - edited I need to state the obvious, because I am a mentor, but while it's good to get this machinery working, this project cannot actually publish any artifacts until its first official release.
          Hide
          Neha Narkhede added a comment -

          Thanks for picking this up. Also, do you mind filing another JIRA for the sbt 0.10.1 upgrade ?

          Show
          Neha Narkhede added a comment - Thanks for picking this up. Also, do you mind filing another JIRA for the sbt 0.10.1 upgrade ?
          Hide
          Joshua Hartman added a comment -

          I did this last night for one of our other projects, Norbert.

          If I upgrade Kafka to sbt 0.10.1, I should be able to do this. Will also write a blog post so other projects know how to do this with maven.

          Show
          Joshua Hartman added a comment - I did this last night for one of our other projects, Norbert. If I upgrade Kafka to sbt 0.10.1, I should be able to do this. Will also write a blog post so other projects know how to do this with maven.

            People

            • Assignee:
              Unassigned
              Reporter:
              Neha Narkhede
            • Votes:
              14 Vote for this issue
              Watchers:
              29 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development