Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.8.1, 0.9.0
    • Fix Version/s: 0.8.1, 0.8.1.1
    • Component/s: packaging
    • Labels:
      None

      Description

      We have previously discussed moving away from SBT to an easier-to-comprehend-and-debug build system such as Ant or Gradle. I put up a patch for an Ant+Ivy build a while ago[1], and it sounded like people wanted to check out Gradle as well.

      1. https://issues.apache.org/jira/browse/KAFKA-855

      1. 0001-Adding-basic-Gradle-build.patch
        76 kB
        David Arthur
      2. 0001-Adding-basic-Gradle-build.patch
        11 kB
        Joe Stein
      3. 0001-Adding-basic-Gradle-build.patch
        68 kB
        David Arthur
      4. 0001-Adding-basic-Gradle-build.patch
        67 kB
        David Arthur
      5. 0001-Adding-basic-Gradle-build.patch
        66 kB
        David Arthur
      6. 0001-Adding-basic-Gradle-build.patch
        66 kB
        David Arthur
      7. 0001-Adding-basic-Gradle-build.patch
        2 kB
        David Arthur
      8. kafka-1171_v10.patch
        78 kB
        Jun Rao
      9. kafka-1171_v11.patch
        79 kB
        Jun Rao
      10. kafka-1171_v12.patch
        79 kB
        Jun Rao
      11. kafka-1171_v13.patch
        81 kB
        Jun Rao
      12. kafka-1171_v14.patch
        81 kB
        Jun Rao
      13. kafka-1171_v15.patch
        82 kB
        Jun Rao
      14. kafka-1171_v6.patch
        73 kB
        Jun Rao
      15. kafka-1171_v7.patch
        75 kB
        Jun Rao
      16. kafka-1171_v8.patch
        75 kB
        Jun Rao
      17. kafka-1171_v9.patch
        74 kB
        Jun Rao

        Issue Links

          Activity

          Hide
          David Arthur added a comment - - edited

          Here is a very simple Gradle project that compiles and tests the "core" part of the Kafka source tree.

          Notable things missing:

          • Multiple versions of Scala (http://issues.gradle.org/browse/GRADLE-2478)
          • Other sub-modules (perf, contrib, etc)
          • Release packaging
          • Gradle wrapper (to avoid developers needing it installed system-wide)
          • Other stuff I'm forgetting at the moment

          As I understand it, all of these things are possible with existing Gradle plugins except for multiple Scala versions - couldn't find an answer there.

          Compared with the Ant build, it is much less verbose, but less flexible. Doing things like cross-targeting Scala is pretty cut-and-dry with Ant. Setting up this simple project up didn't take me very long (~1.5 hours) but I did run into some bad documentation from Gradle's website and some unhelpful errors. Here are two of the more fun errors I encountered: https://gist.github.com/mumrah/6de7be3d6640059904b5

          The error messaging gives me pause, b/c (in my experience) it's almost immediately obvious what went wrong with your build in Ant. Also notice that Gradle is calling the ant scalac target.

          Looking forward hear to what others think!

          Show
          David Arthur added a comment - - edited Here is a very simple Gradle project that compiles and tests the "core" part of the Kafka source tree. Notable things missing: Multiple versions of Scala ( http://issues.gradle.org/browse/GRADLE-2478 ) Other sub-modules (perf, contrib, etc) Release packaging Gradle wrapper (to avoid developers needing it installed system-wide) Other stuff I'm forgetting at the moment As I understand it, all of these things are possible with existing Gradle plugins except for multiple Scala versions - couldn't find an answer there. Compared with the Ant build, it is much less verbose, but less flexible. Doing things like cross-targeting Scala is pretty cut-and-dry with Ant. Setting up this simple project up didn't take me very long (~1.5 hours) but I did run into some bad documentation from Gradle's website and some unhelpful errors. Here are two of the more fun errors I encountered: https://gist.github.com/mumrah/6de7be3d6640059904b5 The error messaging gives me pause, b/c (in my experience) it's almost immediately obvious what went wrong with your build in Ant. Also notice that Gradle is calling the ant scalac target. Looking forward hear to what others think!
          Hide
          David Arthur added a comment -

          Adding patch file instead of diff

          Show
          David Arthur added a comment - Adding patch file instead of diff
          Hide
          Joe Stein added a comment -

          Exciting!!! What do think about using a wrapper like this ( or even just using this wrapper https://github.com/apache/incubator-aurora/blob/master/gradlew) it's nice because it downloads grade for you as not everyone uses brew and on Linux no one does

          Show
          Joe Stein added a comment - Exciting!!! What do think about using a wrapper like this ( or even just using this wrapper https://github.com/apache/incubator-aurora/blob/master/gradlew ) it's nice because it downloads grade for you as not everyone uses brew and on Linux no one does
          Hide
          David Arthur added a comment -

          Joe Stein, yea we should definitely use the wrapper. I think it would also allow us to lock down a particular version of Gradle too.

          Show
          David Arthur added a comment - Joe Stein , yea we should definitely use the wrapper. I think it would also allow us to lock down a particular version of Gradle too.
          Hide
          David Arthur added a comment -

          Created reviewboard https://reviews.apache.org/r/16102/
          against branch origin/trunk

          Show
          David Arthur added a comment - Created reviewboard https://reviews.apache.org/r/16102/ against branch origin/trunk
          Hide
          Jun Rao added a comment -

          David,

          Thanks for the patch. Is that all we need to do to switch to Gradle? Does the patch support maven publishing and ide project build?

          Show
          Jun Rao added a comment - David, Thanks for the patch. Is that all we need to do to switch to Gradle? Does the patch support maven publishing and ide project build?
          Hide
          David Arthur added a comment -

          Jun,

          Gradle does have plugins for IDEs and Maven publishing (as well as many
          others). What it's missing right now is support for building against
          different Scala versions.

          Show
          David Arthur added a comment - Jun, Gradle does have plugins for IDEs and Maven publishing (as well as many others). What it's missing right now is support for building against different Scala versions.
          Hide
          Chris Riccomini added a comment -

          For cross-building Scala versions, have a look at:

          https://issues.apache.org/jira/browse/SAMZA-34

          Szczepan Faber provided the example zip file.

          Show
          Chris Riccomini added a comment - For cross-building Scala versions, have a look at: https://issues.apache.org/jira/browse/SAMZA-34 Szczepan Faber provided the example zip file.
          Hide
          Chris Riccomini added a comment -

          For the record, a good set of examples on how to build OSS multi-module Gradle projects is Netflix's GitHub account:

          https://github.com/Netflix

          Their account contains a number of projects that use Gradle (Lipstick, astyanax, servo, etc). In addition, they have a stub repo for bootstrapping new projects with Gradle.

          https://github.com/Netflix/gradle-template

          Show
          Chris Riccomini added a comment - For the record, a good set of examples on how to build OSS multi-module Gradle projects is Netflix's GitHub account: https://github.com/Netflix Their account contains a number of projects that use Gradle (Lipstick, astyanax, servo, etc). In addition, they have a stub repo for bootstrapping new projects with Gradle. https://github.com/Netflix/gradle-template
          Hide
          David Arthur added a comment -

          Chris Riccomini thanks for the pointer.

          When I try out that sample you have on the SAMZA issue, I get the following:

          
          FAILURE: Build failed with an exception.
          
          * Where:
          Script '/Users/mumrah/Downloads/multi-scala/scala.gradle' line: 44
          
          * What went wrong:
          A problem occurred evaluating root project 'multi-scala'.
          > Failed to notify action.
             > groovy.lang.MissingMethodException: No signature of method: org.gradle.api.internal.artifacts.configurations.DefaultResolutionStrategy.eachDependency() is applicable for argument types: (scala_319e62rbueqtrto35ji6mp2hbc$_configureResolution_closure3) values: [scala_319e62rbueqtrto35ji6mp2hbc$_configureResolution_closure3@17e06b12]
          
          * Try:
          Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
          
          BUILD FAILED
          
          Total time: 4.169 secs
          
          Show
          David Arthur added a comment - Chris Riccomini thanks for the pointer. When I try out that sample you have on the SAMZA issue, I get the following: FAILURE: Build failed with an exception. * Where: Script '/Users/mumrah/Downloads/multi-scala/scala.gradle' line: 44 * What went wrong: A problem occurred evaluating root project 'multi-scala'. > Failed to notify action. > groovy.lang.MissingMethodException: No signature of method: org.gradle.api.internal.artifacts.configurations.DefaultResolutionStrategy.eachDependency() is applicable for argument types: (scala_319e62rbueqtrto35ji6mp2hbc$_configureResolution_closure3) values: [scala_319e62rbueqtrto35ji6mp2hbc$_configureResolution_closure3@17e06b12] * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. BUILD FAILED Total time: 4.169 secs
          Hide
          Chris Riccomini added a comment -

          That sounds a lot like a version incompatibility between the version of Gradle you're using, and what the project was built against (it includes no gradlew script).

          I was able to successfully compile it use Samza's gradlew script. Might setup a gradlew in the project, and give that a shot.

          Show
          Chris Riccomini added a comment - That sounds a lot like a version incompatibility between the version of Gradle you're using, and what the project was built against (it includes no gradlew script). I was able to successfully compile it use Samza's gradlew script. Might setup a gradlew in the project, and give that a shot.
          Hide
          David Arthur added a comment - - edited

          Made some progress with the multi-scala plugin from https://issues.apache.org/jira/browse/SAMZA-34. A few things work, some don't. Top level tasks (assemble, build) don't work, but calling the specific tasks like "./gradlew core:compileScala2_10Java core:compileScala2_10Scala" do work.

          I can't figure out how to override the source sets however, which is important b/c we have Scala-version-dependent code in some places (e.g., Annotations_2.9+.scala).

          After hacking on this for a few hours, I'm inclined to go back to Ant. The inability to debug what is happening under the covers with Gradle is pretty frustrating.

          If we really want to use Gradle for the other plugins, we could create a simple Ant build script for doing the cross-version build and call it from Gradle (http://www.gradle.org/docs/current/userguide/ant.html). However, this feels like asking from trouble (unnecessary complexity).

          What do others think?

          Show
          David Arthur added a comment - - edited Made some progress with the multi-scala plugin from https://issues.apache.org/jira/browse/SAMZA-34 . A few things work, some don't. Top level tasks (assemble, build) don't work, but calling the specific tasks like "./gradlew core:compileScala2_10Java core:compileScala2_10Scala" do work. I can't figure out how to override the source sets however, which is important b/c we have Scala-version-dependent code in some places (e.g., Annotations_2.9+.scala). After hacking on this for a few hours, I'm inclined to go back to Ant. The inability to debug what is happening under the covers with Gradle is pretty frustrating. If we really want to use Gradle for the other plugins, we could create a simple Ant build script for doing the cross-version build and call it from Gradle ( http://www.gradle.org/docs/current/userguide/ant.html ). However, this feels like asking from trouble (unnecessary complexity). What do others think?
          Hide
          David Arthur added a comment -

          Updating patch to include multi-scala plugin and gradlew script

          Show
          David Arthur added a comment - Updating patch to include multi-scala plugin and gradlew script
          Hide
          David Arthur added a comment -

          I dusted off the Ant build I put together a while ago and got it working with the latest trunk. Here is the current state (on my GitHub fork): https://github.com/mumrah/kafka/compare/apache:trunk...ant

          If no one strongly objects (or gets Gradle working), I'm going to continue down the Ant path and update KAFKA-855.

          Show
          David Arthur added a comment - I dusted off the Ant build I put together a while ago and got it working with the latest trunk. Here is the current state (on my GitHub fork): https://github.com/mumrah/kafka/compare/apache:trunk...ant If no one strongly objects (or gets Gradle working), I'm going to continue down the Ant path and update KAFKA-855 .
          Hide
          Joe Stein added a comment -

          what do we gain moving away from SBT to ant? Ant its is not well adopted in the scala community and for a scala project i think that is an important consideration

          Show
          Joe Stein added a comment - what do we gain moving away from SBT to ant? Ant its is not well adopted in the scala community and for a scala project i think that is an important consideration
          Hide
          David Arthur added a comment -

          Joe Stein, I think we would gain plenty, but that's just me.

          What I like about Ant: It's well understood tech - very old, boring, and stable. It has very wide adoption (lots of people know it). Build scripts are easily understood and modified (no "magic"). The biggest downside is that it is pretty verbose and not well loved by the Scala community.

          As for SBT, the only real benefits I see are the cross-building you get for free. Everything else seems like a pain. The DSL is incomprehensible without significant investment (for a build system, this is crazy, IMO). I haven't used it very much, but it seems to be a moving target that is constantly breaking compatibility of plugins.

          As far as I see it:

          • We can make SBT work, but it's caused us pain so far
          • We can maybe make Gradle work, but it's not obvious to me
          • We can make Ant work, but it's new and unpopular

          Back to Gradle for a moment:

          In the Samza build, they parameterize the Scala version which is pretty straightforward. This allows for doing things like ./gradlew -DscalaVersion=2.9.1 build. It solves the issue of targeting multiple versions, but it won't automatically run the build for all versions. If we can live with this, then we can probably get Gradle working. I'll continue tinkering for now.

          Show
          David Arthur added a comment - Joe Stein , I think we would gain plenty, but that's just me. What I like about Ant: It's well understood tech - very old, boring, and stable. It has very wide adoption (lots of people know it). Build scripts are easily understood and modified (no "magic"). The biggest downside is that it is pretty verbose and not well loved by the Scala community. As for SBT, the only real benefits I see are the cross-building you get for free. Everything else seems like a pain. The DSL is incomprehensible without significant investment (for a build system, this is crazy, IMO). I haven't used it very much, but it seems to be a moving target that is constantly breaking compatibility of plugins. As far as I see it: We can make SBT work, but it's caused us pain so far We can maybe make Gradle work, but it's not obvious to me We can make Ant work, but it's new and unpopular Back to Gradle for a moment: In the Samza build, they parameterize the Scala version which is pretty straightforward. This allows for doing things like ./gradlew -DscalaVersion=2.9.1 build . It solves the issue of targeting multiple versions, but it won't automatically run the build for all versions. If we can live with this, then we can probably get Gradle working. I'll continue tinkering for now.
          Hide
          Joe Stein added a comment -

          Even moving to Gradle I will ask the same questions.

          What are we gaining vs the risk involved of ripping something out that is fundamentally working (couple issues that are minor fixes now at this point). Will test cases run faster? Will it be easier for folks to integrate it into their environment when doing development?

          What problems are we trying to solve by introducing this change?

          I have read enough about gradle that I would not be surprised if an argument could be made a proved out it is better than sbt if it all actually worked and could do everything (including publishing repo, building release, etc) ... I feel ant/xml to be very verbose and something that has continually become out of favor which means less people to steward and support projects using it (resources go down).

          Not sure if you have seen this but it seems in gradle you can do ant task type things http://www.gradle.org/docs/current/userguide/ant.html

          Show
          Joe Stein added a comment - Even moving to Gradle I will ask the same questions. What are we gaining vs the risk involved of ripping something out that is fundamentally working (couple issues that are minor fixes now at this point). Will test cases run faster? Will it be easier for folks to integrate it into their environment when doing development? What problems are we trying to solve by introducing this change? I have read enough about gradle that I would not be surprised if an argument could be made a proved out it is better than sbt if it all actually worked and could do everything (including publishing repo, building release, etc) ... I feel ant/xml to be very verbose and something that has continually become out of favor which means less people to steward and support projects using it (resources go down). Not sure if you have seen this but it seems in gradle you can do ant task type things http://www.gradle.org/docs/current/userguide/ant.html
          Hide
          David Arthur added a comment -

          I think having a maintainable build is a big plus. SBT does not lend itself to this. It also seems that Gradle has been gaining traction in the Scala world (probably b/c people dislike SBT so much). Netflix has adopted it, and Samza is using it. Having two closely related projects (Kafka and Samza) use the same build makes good sense.

          And yea, that Ant task thing looks neat.

          Show
          David Arthur added a comment - I think having a maintainable build is a big plus. SBT does not lend itself to this. It also seems that Gradle has been gaining traction in the Scala world (probably b/c people dislike SBT so much). Netflix has adopted it, and Samza is using it. Having two closely related projects (Kafka and Samza) use the same build makes good sense. And yea, that Ant task thing looks neat.
          Hide
          Kostya Golikov added a comment -

          David Arthur talking about sbt pros – it is able to perform incremental compilation and continuous tests execution [out of the box], whereas in ant you are doomed to manual clean/compile loop (aren't you?). It is not vital and showstopper, but such things could ease developer life a lot.

          Show
          Kostya Golikov added a comment - David Arthur talking about sbt pros – it is able to perform incremental compilation and continuous tests execution [out of the box] , whereas in ant you are doomed to manual clean/compile loop (aren't you?). It is not vital and showstopper, but such things could ease developer life a lot.
          Hide
          David Arthur added a comment -

          Kostya Golikov, looks like the Gradle Scala plugin supports Zinc as well as scalac. I'll update my patch if I can get this working.

          Show
          David Arthur added a comment - Kostya Golikov , looks like the Gradle Scala plugin supports Zinc as well as scalac. I'll update my patch if I can get this working.
          Hide
          Jay Kreps added a comment -

          These build discussions tend to blow up due to the bike shed factor

          Here is my two cents:
          1. We migrated off ant to sbt because it seemed shiny. We never really learned to use it properly. No one "owns" it.
          2. SBT has been a pain.
          3. The most important thing is that the committers learn the build system. This will be easier if it is something standard like ant/mvn since the skill is more reusable, but even if we choose something esoteric we just need to suck it up and learn it. Same if we decide to stay with SBT.
          4. LinkedIn uses gradle so for the LinkedIn people that is mainstream though it is not common elsewhere. We also have paid support from the gradle people so we can punt harder questions to them. Because we have consulting from the play people we may actually have access to the same thing for sbt though we never used it due to, I don't know, machisimo.
          5. Either because people won't learn it, can't learn it, or because it can't do it, some of the SBT stuff is suboptimal. For example unit tests don't produce an html unit test report, don't give stack traces, etc. This makes CI less useful.
          6. Whatever we do we cannot regress multi-version scala build or building release artifacts and maven publishing. Ideally we would not regress incremental compilation but I think the IDE does this anyway so that may be good enough (it is for me).
          7. I am willing to live with any technology choice but I think the most important two things to be successful is that (a) someone officially own build and work to make it good and help others when they get stumped (b) everyone learn the technology we choose. Would anyone be willing to do that?

          Show
          Jay Kreps added a comment - These build discussions tend to blow up due to the bike shed factor Here is my two cents: 1. We migrated off ant to sbt because it seemed shiny. We never really learned to use it properly. No one "owns" it. 2. SBT has been a pain. 3. The most important thing is that the committers learn the build system. This will be easier if it is something standard like ant/mvn since the skill is more reusable, but even if we choose something esoteric we just need to suck it up and learn it. Same if we decide to stay with SBT. 4. LinkedIn uses gradle so for the LinkedIn people that is mainstream though it is not common elsewhere. We also have paid support from the gradle people so we can punt harder questions to them. Because we have consulting from the play people we may actually have access to the same thing for sbt though we never used it due to, I don't know, machisimo. 5. Either because people won't learn it, can't learn it, or because it can't do it, some of the SBT stuff is suboptimal. For example unit tests don't produce an html unit test report, don't give stack traces, etc. This makes CI less useful. 6. Whatever we do we cannot regress multi-version scala build or building release artifacts and maven publishing. Ideally we would not regress incremental compilation but I think the IDE does this anyway so that may be good enough (it is for me). 7. I am willing to live with any technology choice but I think the most important two things to be successful is that (a) someone officially own build and work to make it good and help others when they get stumped (b) everyone learn the technology we choose. Would anyone be willing to do that?
          Hide
          David Arthur added a comment - - edited

          Basic parameterized build is working. Added in Zinc compiler for incremental build

          Run it like: ./gradlew -PscalaVersion=2.9.1 clean core:test

          Show
          David Arthur added a comment - - edited Basic parameterized build is working. Added in Zinc compiler for incremental build Run it like: ./gradlew -PscalaVersion=2.9.1 clean core:test
          Hide
          David Arthur added a comment -

          Yea, I agree we're bike-shedding a bit. Any of the options discussed (Ant, Gradle, SBT) can be made to work (probably). To your last point Jay Kreps, I'd be happy to own the build... provided that it's not SBT

          Right now, to me, Gradle looks like the most attractive option.

          Speaking of CI, are we doing anything with Jenkins or TravisCI?

          Show
          David Arthur added a comment - Yea, I agree we're bike-shedding a bit. Any of the options discussed (Ant, Gradle, SBT) can be made to work (probably). To your last point Jay Kreps , I'd be happy to own the build... provided that it's not SBT Right now, to me, Gradle looks like the most attractive option. Speaking of CI, are we doing anything with Jenkins or TravisCI?
          Hide
          Neha Narkhede added a comment -

          +1 on Gradle

          Show
          Neha Narkhede added a comment - +1 on Gradle
          Hide
          David Arthur added a comment -

          Updating patch:

          • Added "perf" subproject
          • Use Zinc for incremental compilation
          • Compile with Scala 2.8.0, 2.8.2, 2.9.1, 2.9.2, 2.10.0
          • Test passing except for 2.10.0
          • Added license checking (taken from Samza)
          • Added Maven publishing, ./gradlew uploadArtifacts see gradle.properties for Maven URL and auth

          Compile and test with "build" target: ./gradlew -PscalaVersion=2.9.2 build

          Show
          David Arthur added a comment - Updating patch: Added "perf" subproject Use Zinc for incremental compilation Compile with Scala 2.8.0, 2.8.2, 2.9.1, 2.9.2, 2.10.0 Test passing except for 2.10.0 Added license checking (taken from Samza) Added Maven publishing, ./gradlew uploadArtifacts see gradle.properties for Maven URL and auth Compile and test with "build" target: ./gradlew -PscalaVersion=2.9.2 build
          Hide
          Joe Stein added a comment -

          Hi David, I applied the latest patch and got stuck after the build target

          Joes-MacBook-Air:kafka joestein$ ./gradlew -PscalaVersion=2.9.2 build
          -bash: ./gradlew: Permission denied
          Joes-MacBook-Air:kafka joestein$ chmod a+x gradlew
          Joes-MacBook-Air:kafka joestein$ ./gradlew -PscalaVersion=2.9.2 build
          Error: Could not find or load main class org.gradle.wrapper.GradleWrapperMain

          let me know if I am missing a step please

          Show
          Joe Stein added a comment - Hi David, I applied the latest patch and got stuck after the build target Joes-MacBook-Air:kafka joestein$ ./gradlew -PscalaVersion=2.9.2 build -bash: ./gradlew: Permission denied Joes-MacBook-Air:kafka joestein$ chmod a+x gradlew Joes-MacBook-Air:kafka joestein$ ./gradlew -PscalaVersion=2.9.2 build Error: Could not find or load main class org.gradle.wrapper.GradleWrapperMain let me know if I am missing a step please
          Hide
          Jun Rao added a comment -

          Thanks for the patch. I tried ./gradlew build and got the following error.

          FAILURE: Build failed with an exception.

          • Where:
            Build file '/Users/jrao/Intellij/kafka_git/build.gradle' line: 5
          • What went wrong:
            A problem occurred evaluating root project 'kafka_git'.
            > Could not read script '/Users/jrao/Intellij/kafka_git/gradle/buildscript.gradle' as it does not exist.

          Also, could you describe how to the following:
          1. ide support
          2. build a release tar/zip
          3. run a single unit test (the test-only option in sbt)
          4. Is there still a assembly-package-dependency step?

          Show
          Jun Rao added a comment - Thanks for the patch. I tried ./gradlew build and got the following error. FAILURE: Build failed with an exception. Where: Build file '/Users/jrao/Intellij/kafka_git/build.gradle' line: 5 What went wrong: A problem occurred evaluating root project 'kafka_git'. > Could not read script '/Users/jrao/Intellij/kafka_git/gradle/buildscript.gradle' as it does not exist. Also, could you describe how to the following: 1. ide support 2. build a release tar/zip 3. run a single unit test (the test-only option in sbt) 4. Is there still a assembly-package-dependency step?
          Hide
          David Arthur added a comment -

          Updating patch to include two missing .gradle files

          Show
          David Arthur added a comment - Updating patch to include two missing .gradle files
          Hide
          David Arthur added a comment -

          Sorry for being quiet on this for a few weeks - just getting back into working mode after the holidays.

          Joe Stein, still having permissions issues? Does something need a chmod +x?

          Jun Rao, to answer your questions:

          1. Gradle will generate IDEA and Eclipse project files (I've tested this a little)
          2. Building a release is a matter of writing a Gradle task that does all the necessary things (build jar, build javadocs, copy bin scripts, etc). There are some Gradle plugins to help with some tasks (e.g., license checking)
          3.

          {./gradlew -Dtest.single=RequestResponseSerializationTest core:test}

          4. Not sure - what is the purpose of this step in the current build? If it's for gathering deps for a distribution? I see this Gradle plugin (http://www.gradle.org/docs/current/userguide/distribution_plugin.html) which I haven't tried yet.

          Show
          David Arthur added a comment - Sorry for being quiet on this for a few weeks - just getting back into working mode after the holidays. Joe Stein , still having permissions issues? Does something need a chmod +x? Jun Rao , to answer your questions: 1. Gradle will generate IDEA and Eclipse project files (I've tested this a little) 2. Building a release is a matter of writing a Gradle task that does all the necessary things (build jar, build javadocs, copy bin scripts, etc). There are some Gradle plugins to help with some tasks (e.g., license checking) 3. {./gradlew -Dtest.single=RequestResponseSerializationTest core:test} 4. Not sure - what is the purpose of this step in the current build? If it's for gathering deps for a distribution? I see this Gradle plugin ( http://www.gradle.org/docs/current/userguide/distribution_plugin.html ) which I haven't tried yet.
          Hide
          Joe Stein added a comment -

          with the new patch I was able to get

          ./gradlew -PscalaVersion=2.8.0 clean core:test
          ./gradlew -PscalaVersion=2.8.2 clean core:test
          ./gradlew -PscalaVersion=2.9.1 clean core:test
          ./gradlew -PscalaVersion=2.9.2 clean core:test

          to BUILD SUCCESSFUL

          however

          ./gradlew -PscalaVersion=2.10.1 clean core:test

          fails with java.lang.NoClassDefFoundError for every test (same trying 2.10.2 and 2.10.3)

          are there other tasks to try out yet or not yet? Would be cool to have the README updated/changed also to reflect the gradle tasks that are supported / pending for everything to switch the existing README out from sbt to gradle (once everything is in there of course).

          Show
          Joe Stein added a comment - with the new patch I was able to get ./gradlew -PscalaVersion=2.8.0 clean core:test ./gradlew -PscalaVersion=2.8.2 clean core:test ./gradlew -PscalaVersion=2.9.1 clean core:test ./gradlew -PscalaVersion=2.9.2 clean core:test to BUILD SUCCESSFUL however ./gradlew -PscalaVersion=2.10.1 clean core:test fails with java.lang.NoClassDefFoundError for every test (same trying 2.10.2 and 2.10.3) are there other tasks to try out yet or not yet? Would be cool to have the README updated/changed also to reflect the gradle tasks that are supported / pending for everything to switch the existing README out from sbt to gradle (once everything is in there of course).
          Hide
          Christopher Freeman added a comment -

          In the patch, the scalatest dependency is wrong. In :core dependencies, I replace it with

              if (scalaVersion.startsWith('2.8')) {
                testCompile 'org.scalatest:scalatest:1.2'
              } else if (scalaVersion.startsWith('2.10')) {
                testCompile 'org.scalatest:scalatest_2.10:1.9.1'
              } else {
                testCompile "org.scalatest:scalatest_$scalaVersion:1.8"
              }   
          

          to match the logic in core/build.sbt and ./gradlew -PscalaVersion=2.10.1 now works

          Show
          Christopher Freeman added a comment - In the patch, the scalatest dependency is wrong. In :core dependencies, I replace it with if (scalaVersion.startsWith('2.8')) { testCompile 'org.scalatest:scalatest:1.2' } else if (scalaVersion.startsWith('2.10')) { testCompile 'org.scalatest:scalatest_2.10:1.9.1' } else { testCompile "org.scalatest:scalatest_$scalaVersion:1.8" } to match the logic in core/build.sbt and ./gradlew -PscalaVersion=2.10.1 now works
          Hide
          Guozhang Wang added a comment -

          Downloaded the latest patch and apply on trunk. Some observations:

          1. Need to manually chmod u+x gradlew file.
          2. When run ./gradlew, got the following error:

          Exception in thread "main" java.lang.NoClassDefFoundError: org/gradle/wrapper/GradleWrapperMain
          Caused by: java.lang.ClassNotFoundException: org.gradle.wrapper.GradleWrapperMain
          at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
          at java.security.AccessController.doPrivileged(Native Method)
          at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
          at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
          Could not find the main class: org.gradle.wrapper.GradleWrapperMain. Program will exit.

          Show
          Guozhang Wang added a comment - Downloaded the latest patch and apply on trunk. Some observations: 1. Need to manually chmod u+x gradlew file. 2. When run ./gradlew, got the following error: Exception in thread "main" java.lang.NoClassDefFoundError: org/gradle/wrapper/GradleWrapperMain Caused by: java.lang.ClassNotFoundException: org.gradle.wrapper.GradleWrapperMain at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) Could not find the main class: org.gradle.wrapper.GradleWrapperMain. Program will exit.
          Hide
          Jun Rao added a comment -

          I made a pass of this. Attach the latest patch v6, which includes:
          1. added a new task for copying all dependent jars locally and fixed Kafka script to pick up those jars
          2. added a new task for building the release tar gz file
          3. fixed the name of the built artifacts (with kafka prefix and scala version)
          4. fixed READE with the new commands.
          5. added the license HEADER for build to work.

          TODO:
          1. Currently, we default to build with scala 2.9.1. So the script only works if you build by specifying scala 2.8.0. Will look into how to default to build with scala 2.8.0.
          2. Task "idea" fails with:
          Execution failed for task ':ideaProject'.
          > Cannot infer Scala class path because no Scala library Jar was found on class path: [/Users/jrao/.gradle/caches/artifacts-24/filestore/net.sf.jopt-simple/jopt-simple/3.2/jar/d625f12ba08083c8c16dcedd5396ec730e9e77ab/jopt-simple-3.2.jar]
          3. There is a slight difference in artifact names. Currently, we have
          kafka_2.8.0-0.8.1.jar
          With this patch, it's changed to
          kafka-2.8.0-0.8.1.jar
          Will look into how to fix it.

          Show
          Jun Rao added a comment - I made a pass of this. Attach the latest patch v6, which includes: 1. added a new task for copying all dependent jars locally and fixed Kafka script to pick up those jars 2. added a new task for building the release tar gz file 3. fixed the name of the built artifacts (with kafka prefix and scala version) 4. fixed READE with the new commands. 5. added the license HEADER for build to work. TODO: 1. Currently, we default to build with scala 2.9.1. So the script only works if you build by specifying scala 2.8.0. Will look into how to default to build with scala 2.8.0. 2. Task "idea" fails with: Execution failed for task ':ideaProject'. > Cannot infer Scala class path because no Scala library Jar was found on class path: [/Users/jrao/.gradle/caches/artifacts-24/filestore/net.sf.jopt-simple/jopt-simple/3.2/jar/d625f12ba08083c8c16dcedd5396ec730e9e77ab/jopt-simple-3.2.jar] 3. There is a slight difference in artifact names. Currently, we have kafka_2.8.0-0.8.1.jar With this patch, it's changed to kafka-2.8.0-0.8.1.jar Will look into how to fix it.
          Hide
          Jun Rao added a comment -

          Joe Stein,

          Do you think you can take a look at maven publishing? I tried to publish locally using mavenLocal(). However, I didn't see any pom file generated in the .m2/ (only saw the jar and the ivy file). Is that normal?

          Show
          Jun Rao added a comment - Joe Stein, Do you think you can take a look at maven publishing? I tried to publish locally using mavenLocal(). However, I didn't see any pom file generated in the .m2/ (only saw the jar and the ivy file). Is that normal?
          Hide
          Joe Stein added a comment -

          yup, will take a look.

          Show
          Joe Stein added a comment - yup, will take a look.
          Hide
          Joe Stein added a comment -

          patched the build to sign the jars, the pom during uploadArchives

          stage passes with sigs but pom is blank https://repository.apache.org/content/groups/staging/org/apache/kafka/kafka_2.9.1/0.8.1/kafka_2.9.1-0.8.1.pom will look into that next

          Show
          Joe Stein added a comment - patched the build to sign the jars, the pom during uploadArchives stage passes with sigs but pom is blank https://repository.apache.org/content/groups/staging/org/apache/kafka/kafka_2.9.1/0.8.1/kafka_2.9.1-0.8.1.pom will look into that next
          Hide
          Jun Rao added a comment -

          Attaching v6 patch again with the gradle binary jar included.

          Joe,

          Could you add your change on top of v6? When generating the patch, make sure you use the "--binary" in git diff so that the binary jar is included.

          Show
          Jun Rao added a comment - Attaching v6 patch again with the gradle binary jar included. Joe, Could you add your change on top of v6? When generating the patch, make sure you use the "--binary" in git diff so that the binary jar is included.
          Hide
          David Arthur added a comment - - edited

          Figured out the POM thing finally.

          After Google utterly failed me, I started looking at Netflix OSS (big Gradle users) and see what they are doing different. I came across the following tidbit in Archaius https://github.com/Netflix/archaius/blob/master/gradle/maven.gradle#L3

          apply plugin: 'maven' // Java plugin has to have been already applied for the conf2scope mappings to work
          

          I added

          apply plugin: 'java'
          

          to the subprojects declaration and now I see a fully populated POM in my Nexus repo. My local state is pretty messed up at the moment, so I'll have to wait until tomorrow evening to generate a new patch - or someone else can do it.

          One step closer...

          Show
          David Arthur added a comment - - edited Figured out the POM thing finally. After Google utterly failed me, I started looking at Netflix OSS (big Gradle users) and see what they are doing different. I came across the following tidbit in Archaius https://github.com/Netflix/archaius/blob/master/gradle/maven.gradle#L3 apply plugin: 'maven' // Java plugin has to have been already applied for the conf2scope mappings to work I added apply plugin: 'java' to the subprojects declaration and now I see a fully populated POM in my Nexus repo. My local state is pretty messed up at the moment, so I'll have to wait until tomorrow evening to generate a new patch - or someone else can do it. One step closer...
          Hide
          Jun Rao added a comment -

          Attach patch v7. Additionally fixes since v6.
          1. defaulted the build to scala 2.8.0.
          2. fixed the build artifact names to be consistent with those produced in sbt.
          3. added the task jar_all to build the jar for all scala versions.
          4. fixed the runtime dependency; now both copyDependantLibs and releaseTarGz generate the correct runtime dependent jars.
          5. included David's fix on maven; pom file can be generated now.
          6. fixed README.md.

          TODO:
          1. Task "idea" fails with:
          Execution failed for task ':ideaProject'.
          > Cannot infer Scala class path because no Scala library Jar was found on class path: [/Users/jrao/.gradle/caches/artifacts-24/filestore/net.sf.jopt-simple/jopt-simple/3.2/jar/d625f12ba08083c8c16dcedd5396ec730e9e77ab/jopt-simple-3.2.jar]
          2. The dependencies listed in pom file is a bit different from runtime dependencies.

          The dependencies in the pom file (./gradlew :core:uploadArchives) include:
          <artifactId>easymock</artifactId>
          <artifactId>jopt-simple</artifactId>
          <artifactId>junit</artifactId>
          <artifactId>metrics-annotation</artifactId>
          <artifactId>metrics-core</artifactId>
          <artifactId>objenesis</artifactId>
          <artifactId>scala-library</artifactId>
          <artifactId>scalatest</artifactId>
          <artifactId>snappy-java</artifactId>
          <artifactId>zkclient</artifactId>
          <artifactId>zookeeper</artifactId>

          Runtime dependencies (./gradlew copyDependantLibs) are:
          ./core/build/dependant-libs-2.8.0/jopt-simple-3.2.jar
          ./core/build/dependant-libs-2.8.0/log4j-1.2.15.jar
          ./core/build/dependant-libs-2.8.0/metrics-annotation-2.2.0.jar
          ./core/build/dependant-libs-2.8.0/metrics-core-2.2.0.jar
          ./core/build/dependant-libs-2.8.0/scala-library-2.8.0.jar
          ./core/build/dependant-libs-2.8.0/slf4j-api-1.7.2.jar
          ./core/build/dependant-libs-2.8.0/snappy-java-1.0.4.1.jar
          ./core/build/dependant-libs-2.8.0/zkclient-0.3.jar
          ./core/build/dependant-libs-2.8.0/zookeeper-3.3.4.jar

          Show
          Jun Rao added a comment - Attach patch v7. Additionally fixes since v6. 1. defaulted the build to scala 2.8.0. 2. fixed the build artifact names to be consistent with those produced in sbt. 3. added the task jar_all to build the jar for all scala versions. 4. fixed the runtime dependency; now both copyDependantLibs and releaseTarGz generate the correct runtime dependent jars. 5. included David's fix on maven; pom file can be generated now. 6. fixed README.md. TODO: 1. Task "idea" fails with: Execution failed for task ':ideaProject'. > Cannot infer Scala class path because no Scala library Jar was found on class path: [/Users/jrao/.gradle/caches/artifacts-24/filestore/net.sf.jopt-simple/jopt-simple/3.2/jar/d625f12ba08083c8c16dcedd5396ec730e9e77ab/jopt-simple-3.2.jar] 2. The dependencies listed in pom file is a bit different from runtime dependencies. The dependencies in the pom file (./gradlew :core:uploadArchives) include: <artifactId>easymock</artifactId> <artifactId>jopt-simple</artifactId> <artifactId>junit</artifactId> <artifactId>metrics-annotation</artifactId> <artifactId>metrics-core</artifactId> <artifactId>objenesis</artifactId> <artifactId>scala-library</artifactId> <artifactId>scalatest</artifactId> <artifactId>snappy-java</artifactId> <artifactId>zkclient</artifactId> <artifactId>zookeeper</artifactId> Runtime dependencies (./gradlew copyDependantLibs) are: ./core/build/dependant-libs-2.8.0/jopt-simple-3.2.jar ./core/build/dependant-libs-2.8.0/log4j-1.2.15.jar ./core/build/dependant-libs-2.8.0/metrics-annotation-2.2.0.jar ./core/build/dependant-libs-2.8.0/metrics-core-2.2.0.jar ./core/build/dependant-libs-2.8.0/scala-library-2.8.0.jar ./core/build/dependant-libs-2.8.0/slf4j-api-1.7.2.jar ./core/build/dependant-libs-2.8.0/snappy-java-1.0.4.1.jar ./core/build/dependant-libs-2.8.0/zkclient-0.3.jar ./core/build/dependant-libs-2.8.0/zookeeper-3.3.4.jar
          Hide
          Jun Rao added a comment -

          Attach patch v8.

          1. ./gradlew idea works now. I still couldn't get it compile in Intellij. However, this is due to an existing problem that we have two versions of Annotations.scala (one for 2.8 and another for 2.9+).

          TODO:
          1. The dependencies in the pom file is still a bit weird. Unlike runtime dependencies, pom only includes direct dependencies. However, it's not clear why the following dependencies are listed since they are test dependencies only.
          'org.easymock:easymock:3.0'
          'org.objenesis:objenesis:1.2'
          'org.scalatest:scalatest:1.2'

          Show
          Jun Rao added a comment - Attach patch v8. 1. ./gradlew idea works now. I still couldn't get it compile in Intellij. However, this is due to an existing problem that we have two versions of Annotations.scala (one for 2.8 and another for 2.9+). TODO: 1. The dependencies in the pom file is still a bit weird. Unlike runtime dependencies, pom only includes direct dependencies. However, it's not clear why the following dependencies are listed since they are test dependencies only. 'org.easymock:easymock:3.0' 'org.objenesis:objenesis:1.2' 'org.scalatest:scalatest:1.2'
          Hide
          Jay Kreps added a comment -

          Shouldn't we default to scala 2.9 or something newer than 2.8?

          Show
          Jay Kreps added a comment - Shouldn't we default to scala 2.9 or something newer than 2.8?
          Hide
          Guozhang Wang added a comment -

          Used Linux to try the patch, still get the following exception on running ./gradlew:

          Exception in thread "main" java.lang.NoClassDefFoundError: org/gradle/wrapper/GradleWrapperMain
          Caused by: java.lang.ClassNotFoundException: org.gradle.wrapper.GradleWrapperMain
          at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
          at java.security.AccessController.doPrivileged(Native Method)
          at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
          at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
          Could not find the main class: org.gradle.wrapper.GradleWrapperMain. Program will exit.

          Also, could we avoid manually "chmod u+x" on the gradlew file?

          Show
          Guozhang Wang added a comment - Used Linux to try the patch, still get the following exception on running ./gradlew: Exception in thread "main" java.lang.NoClassDefFoundError: org/gradle/wrapper/GradleWrapperMain Caused by: java.lang.ClassNotFoundException: org.gradle.wrapper.GradleWrapperMain at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) Could not find the main class: org.gradle.wrapper.GradleWrapperMain. Program will exit. Also, could we avoid manually "chmod u+x" on the gradlew file?
          Hide
          Jun Rao added a comment -

          Guozhang,

          That class is included in the patch in gradle/wrapper/gradle-wrapper.jar. Do you see it?

          Show
          Jun Rao added a comment - Guozhang, That class is included in the patch in gradle/wrapper/gradle-wrapper.jar. Do you see it?
          Hide
          Guozhang Wang added a comment -

          No, when I applied the patch these are the files added:

          [guwang@guwang-ld2 kafka_1171v8]$ patch -p1 < kafka-1171_v8.patch
          patching file HEADER
          patching file README.md
          patching file bin/kafka-run-class.sh
          patching file build.gradle
          patching file gradle.properties
          patching file gradle/buildscript.gradle
          patching file gradle/license.gradle
          patching file gradle/wrapper/gradle-wrapper.properties
          patching file gradlew
          patching file settings.gradle

          ---------

          And the jar file is not included:

          [guwang@guwang-ld2 kafka_1171v8]$ ls gradle/wrapper/
          gradle-wrapper.properties

          Show
          Guozhang Wang added a comment - No, when I applied the patch these are the files added: [guwang@guwang-ld2 kafka_1171v8] $ patch -p1 < kafka-1171_v8.patch patching file HEADER patching file README.md patching file bin/kafka-run-class.sh patching file build.gradle patching file gradle.properties patching file gradle/buildscript.gradle patching file gradle/license.gradle patching file gradle/wrapper/gradle-wrapper.properties patching file gradlew patching file settings.gradle --------- And the jar file is not included: [guwang@guwang-ld2 kafka_1171v8] $ ls gradle/wrapper/ gradle-wrapper.properties
          Hide
          Jun Rao added a comment -

          Attach patch v9. The pom dependency seems to work since the included test dependencies are marked with test.

          1. Added groovy script to simplify cross compile for jar_all and build_all.

          There are no known existing issues. Please verify.

          Joe Stein, could you verify the pom file?

          Show
          Jun Rao added a comment - Attach patch v9. The pom dependency seems to work since the included test dependencies are marked with test. 1. Added groovy script to simplify cross compile for jar_all and build_all. There are no known existing issues. Please verify. Joe Stein, could you verify the pom file?
          Hide
          Jun Rao added a comment -

          Please use "git apply kafka-1171_v9.patch" to try the patch.

          Show
          Jun Rao added a comment - Please use "git apply kafka-1171_v9.patch" to try the patch.
          Hide
          Guozhang Wang added a comment -

          Tried v9 with all the functionalities, a few observations:


          1. When apply the patch some warnings are shown.

          [guwang@guwang-ld2 kafka_1171v8]$ git apply kafka-1171_v9.patch
          kafka-1171_v9.patch:168: trailing whitespace.

          kafka-1171_v9.patch:286: trailing whitespace.
          tasks.create(name: "releaseTarGz", dependsOn: configurations.archives.artifacts, type: Tar) {
          kafka-1171_v9.patch:288: trailing whitespace.
          compression = Compression.GZIP
          warning: 3 lines add whitespace errors.

          -------
          2. The first time I run ./gradlew build all tests passed OK, then if I change one line in one of the test (testCircularIterator) to let it fail, sometimes another test fails also:

          kafka.server.LogRecoveryTest > testHWCheckpointNoFailuresMultipleLogSegments FAILED
          java.lang.AssertionError at LogRecoveryTest.scala:182

          kafka.utils.UtilsTest > testCircularIterator FAILED
          java.lang.AssertionError at UtilsTest.scala:48

          This testHWCheckpointNoFailuresMultipleLogSegments seems have transient failures. I tried the same pattern 3 times and this test failed twice.

          Show
          Guozhang Wang added a comment - Tried v9 with all the functionalities, a few observations: 1. When apply the patch some warnings are shown. [guwang@guwang-ld2 kafka_1171v8] $ git apply kafka-1171_v9.patch kafka-1171_v9.patch:168: trailing whitespace. kafka-1171_v9.patch:286: trailing whitespace. tasks.create(name: "releaseTarGz", dependsOn: configurations.archives.artifacts, type: Tar) { kafka-1171_v9.patch:288: trailing whitespace. compression = Compression.GZIP warning: 3 lines add whitespace errors. ------- 2. The first time I run ./gradlew build all tests passed OK, then if I change one line in one of the test (testCircularIterator) to let it fail, sometimes another test fails also: kafka.server.LogRecoveryTest > testHWCheckpointNoFailuresMultipleLogSegments FAILED java.lang.AssertionError at LogRecoveryTest.scala:182 kafka.utils.UtilsTest > testCircularIterator FAILED java.lang.AssertionError at UtilsTest.scala:48 This testHWCheckpointNoFailuresMultipleLogSegments seems have transient failures. I tried the same pattern 3 times and this test failed twice.
          Hide
          David Arthur added a comment -

          I've seen some transient test failures as well, can't remember off hand what they are though. Once the build is finalized, I think a good next step would be to get some CI going.

          Jun Rao, are there any more outstanding issues?

          Show
          David Arthur added a comment - I've seen some transient test failures as well, can't remember off hand what they are though. Once the build is finalized, I think a good next step would be to get some CI going. Jun Rao , are there any more outstanding issues?
          Hide
          Jun Rao added a comment -

          David,

          I don't think there is any outstanding issue. Could you help check the pom file?

          Show
          Jun Rao added a comment - David, I don't think there is any outstanding issue. Could you help check the pom file?
          Hide
          Jun Rao added a comment -

          Jay,

          Is there any benefit of not defaulting to 2.8.0? I was thinking that we should probably stay with the lowest supported scala version as the default.

          Show
          Jun Rao added a comment - Jay, Is there any benefit of not defaulting to 2.8.0? I was thinking that we should probably stay with the lowest supported scala version as the default.
          Hide
          Jay Kreps added a comment -

          My impression is that people who use Scala have mostly moved on. People who don't use Scala should actually use 2.9 as its supposed to be a bit faster.

          Show
          Jay Kreps added a comment - My impression is that people who use Scala have mostly moved on. People who don't use Scala should actually use 2.9 as its supposed to be a bit faster.
          Hide
          David Arthur added a comment -
          • Adding upload_* tasks for the different Scala versions
          • Add subproject name to the upload artifact
          • Tested uploading all artifacts to Nexus
          • Tested pulling down artifacts via Ivy
          • Fixed javadoc error (remove package.html from util dir)

          Still seeing a few issues (non-blockers):

          • Gradle warnings
          The TaskContainer.add() method has been deprecated and is scheduled to be removed in Gradle 2.0. Please use the create() method instead.
          Building project 'core' with Scala version 2.8.0
          The scalaTools configuration has been deprecated and is scheduled to be removed in Gradle 2.0. Typically, usages of 'scalaTools' can simply be removed, and the Scala tools libraries to be used will be inferred from the Scala library found on the regular (compile) class path. In some cases, it may be necessary to additionally configure the 'scalaClasspath' property of ScalaCompile and ScalaDoc tasks.
          
          • Some of our custom tasks don't show up in the "tasks" output

          Other questions:

          • Publishing docs/sources jars?
          • How to produce a release dist with different Scala versions? Right now it picks 2.8.0, maybe this should default to our "recommended" Scala version since this is probably how people will run Kafka brokers.
          Show
          David Arthur added a comment - Adding upload_* tasks for the different Scala versions Add subproject name to the upload artifact Tested uploading all artifacts to Nexus Tested pulling down artifacts via Ivy Fixed javadoc error (remove package.html from util dir) Still seeing a few issues (non-blockers): Gradle warnings The TaskContainer.add() method has been deprecated and is scheduled to be removed in Gradle 2.0. Please use the create() method instead. Building project 'core' with Scala version 2.8.0 The scalaTools configuration has been deprecated and is scheduled to be removed in Gradle 2.0. Typically, usages of 'scalaTools' can simply be removed, and the Scala tools libraries to be used will be inferred from the Scala library found on the regular (compile) class path. In some cases, it may be necessary to additionally configure the 'scalaClasspath' property of ScalaCompile and ScalaDoc tasks. Some of our custom tasks don't show up in the "tasks" output Other questions: Publishing docs/sources jars? How to produce a release dist with different Scala versions? Right now it picks 2.8.0, maybe this should default to our "recommended" Scala version since this is probably how people will run Kafka brokers.
          Hide
          Jun Rao added a comment -

          Attach patch v10.

          1. added uploadCoreArchives_all task
          2. added releaseTarGz_all task
          3. updated README
          4. added Apache license header
          5. rebased and made fixes to use the new version of snappy.

          David Arthur,

          I didn't incorporate your latest changes. Could you apply what's not already covered on top of v10 and upload a new patch? Thanks.

          Show
          Jun Rao added a comment - Attach patch v10. 1. added uploadCoreArchives_all task 2. added releaseTarGz_all task 3. updated README 4. added Apache license header 5. rebased and made fixes to use the new version of snappy. David Arthur, I didn't incorporate your latest changes. Could you apply what's not already covered on top of v10 and upload a new patch? Thanks.
          Hide
          Jun Rao added a comment -

          Attach patch v11.

          1. Included all but one fixes from David Arthur on top of v9.
          2. Added a new project "clients" for the new producer. To build clients jar and run unit tests, do
          ./gradlew clients:jar
          ./gradlew clients:test
          3. Removed scalaTools from build.gradle to get rid of the warning.
          4. Updated README.md.

          The patch is ready to be checked in (will remove sbt stuff during checking in).

          David Arthur,
          I didn't take your following change since it publishes the kafka jar under kafka-core, which is different from the 0.8.0 release. I haven't figured out how to use the modified project name, but will do. However, this is not a blocker since we only publish one jar to maven now.
          > + pom.artifactId = "kafka-$

          {project.name}

          _$

          {scalaVersion}

          "

          Show
          Jun Rao added a comment - Attach patch v11. 1. Included all but one fixes from David Arthur on top of v9. 2. Added a new project "clients" for the new producer. To build clients jar and run unit tests, do ./gradlew clients:jar ./gradlew clients:test 3. Removed scalaTools from build.gradle to get rid of the warning. 4. Updated README.md. The patch is ready to be checked in (will remove sbt stuff during checking in). David Arthur, I didn't take your following change since it publishes the kafka jar under kafka-core, which is different from the 0.8.0 release. I haven't figured out how to use the modified project name, but will do. However, this is not a blocker since we only publish one jar to maven now. > + pom.artifactId = "kafka-$ {project.name} _$ {scalaVersion} "
          Hide
          David Arthur added a comment - - edited

          Jun, that's fine. I added the subproject name because it was causing a conflict when publishing to Nexus since our artifact pattern is just kafka-${scalaVersion} - so "perf" and "core" were producing the same artifact. It does appear that we publish "kafka", "kafka-perf", and "contrib" artifacts to Maven Central.

          If keeping the name consistent with 0.8.0 is important, then we can do some scripting in the build to achieve this, but otherwise it's simpler to add the subproject name to the artifact.

          On a related note, are there plans to split the artifacts up into core, server, and client? Maybe after Jay Kreps's client refactoring?

          Show
          David Arthur added a comment - - edited Jun, that's fine. I added the subproject name because it was causing a conflict when publishing to Nexus since our artifact pattern is just kafka-${scalaVersion} - so "perf" and "core" were producing the same artifact. It does appear that we publish "kafka", "kafka-perf", and "contrib" artifacts to Maven Central. If keeping the name consistent with 0.8.0 is important, then we can do some scripting in the build to achieve this, but otherwise it's simpler to add the subproject name to the artifact. On a related note, are there plans to split the artifacts up into core, server, and client? Maybe after Jay Kreps 's client refactoring?
          Hide
          Neha Narkhede added a comment -

          Thanks for the patch, Jun.

          1. ./gradlew copyDependantLibs
          Is it a standard gradle command or did we create it? If we created it, can we rename it to assembleDependencies?
          2. Random nitpick: Why is everything ./gradlew and not simply ./gradle?
          3. Most unit tests fail on Scala 2.10.x. These pass on sbt though (./sbt "++2.10.0 test")

          Show
          Neha Narkhede added a comment - Thanks for the patch, Jun. 1. ./gradlew copyDependantLibs Is it a standard gradle command or did we create it? If we created it, can we rename it to assembleDependencies? 2. Random nitpick: Why is everything ./gradlew and not simply ./gradle? 3. Most unit tests fail on Scala 2.10.x. These pass on sbt though (./sbt "++2.10.0 test")
          Hide
          Jun Rao added a comment -

          The unit tests issue for scala 2.10.x is caused by scalatest version. In sbt, we have the following
          libraryDependencies <<= (scalaVersion, libraryDependencies) { (sv, deps) =>
          deps :+ (sv match

          { case "2.8.0" => "org.scalatest" % "scalatest" % "1.2" % "test" case v if v.startsWith("2.10") => "org.scalatest" %% "scalatest" % "1.9.1" % "test" case _ => "org.scalatest" %% "scalatest" % "1.8" % "test" }

          )
          }

          Need to figure out how to add the correct scalatest dependency in gradle.

          Show
          Jun Rao added a comment - The unit tests issue for scala 2.10.x is caused by scalatest version. In sbt, we have the following libraryDependencies <<= (scalaVersion, libraryDependencies) { (sv, deps) => deps :+ (sv match { case "2.8.0" => "org.scalatest" % "scalatest" % "1.2" % "test" case v if v.startsWith("2.10") => "org.scalatest" %% "scalatest" % "1.9.1" % "test" case _ => "org.scalatest" %% "scalatest" % "1.8" % "test" } ) } Need to figure out how to add the correct scalatest dependency in gradle.
          Hide
          Jun Rao added a comment -

          Did a google search on scalatest and found the answer right in this jira. Apparently, Chris Freeman has posted the answer earlier. Attach patch v12.

          1. Unit tests should work with scala 2.10* now.
          2. Added the project example.

          Todo:
          1. Need to figure out how to add the two subprojects under contrib.

          Show
          Jun Rao added a comment - Did a google search on scalatest and found the answer right in this jira. Apparently, Chris Freeman has posted the answer earlier. Attach patch v12. 1. Unit tests should work with scala 2.10* now. 2. Added the project example. Todo: 1. Need to figure out how to add the two subprojects under contrib.
          Hide
          Jun Rao added a comment -

          Attach patch v13. This fixes all known issues.

          1. Added hadoop-producer and hadoop-consumer projects.
          2. Fixed maven publishing to use project name. kafka.jar is still published under kafka, same as we have in maven now. Other projects like hadoop-consumer will now be called kafka-hadoop-consumer to make it clear. Since those jars are not widely used, changing the name is probably ok.
          3. Renamed the *_all tasks a bit and the new tasks are summarized in README.
          4. For copyDependantLibs, I kept the name. The old assemblyDependency target in sbt builds a fat jar. Here, the jars are copied individually and not assembled together, which is clearer.
          5. ./gradlew is probably used to distinguish it from the locally install gradle executable.

          Future stuff:
          1. I think we should keep the core jar name kafka until the old producer/consumer code is phased out, at which point, we can rename it to kafka-server. Before that, since it has the old client, we should leave it in the old coordinate so that an application can dedup the jars correctly if multiple versions of kafka jar are dragged in through transitive dependencies.

          Show
          Jun Rao added a comment - Attach patch v13. This fixes all known issues. 1. Added hadoop-producer and hadoop-consumer projects. 2. Fixed maven publishing to use project name. kafka.jar is still published under kafka, same as we have in maven now. Other projects like hadoop-consumer will now be called kafka-hadoop-consumer to make it clear. Since those jars are not widely used, changing the name is probably ok. 3. Renamed the *_all tasks a bit and the new tasks are summarized in README. 4. For copyDependantLibs, I kept the name. The old assemblyDependency target in sbt builds a fat jar. Here, the jars are copied individually and not assembled together, which is clearer. 5. ./gradlew is probably used to distinguish it from the locally install gradle executable. Future stuff: 1. I think we should keep the core jar name kafka until the old producer/consumer code is phased out, at which point, we can rename it to kafka-server. Before that, since it has the old client, we should leave it in the old coordinate so that an application can dedup the jars correctly if multiple versions of kafka jar are dragged in through transitive dependencies.
          Hide
          Guozhang Wang added a comment -

          Wondering how to run a single unit test:

          ./gradlew -Dtest.single=RequestResponseSerializationTest core:test

          would execute all the tests in cores.

          ./gradlew -Dtest.single=RequestResponseSerializationTest core:test:[SpecificTestName]

          seems not the correct syntax.

          Show
          Guozhang Wang added a comment - Wondering how to run a single unit test: ./gradlew -Dtest.single=RequestResponseSerializationTest core:test would execute all the tests in cores. ./gradlew -Dtest.single=RequestResponseSerializationTest core:test: [SpecificTestName] seems not the correct syntax.
          Hide
          Jun Rao added a comment -

          Attach patch v14 to fix a typo in README.

          Show
          Jun Rao added a comment - Attach patch v14 to fix a typo in README.
          Hide
          Jun Rao added a comment -

          Guozhang,

          I tried
          ./gradlew -Dtest.single=RequestResponseSerializationTest core:test

          and it only ran 1 test.

          Show
          Jun Rao added a comment - Guozhang, I tried ./gradlew -Dtest.single=RequestResponseSerializationTest core:test and it only ran 1 test.
          Hide
          Guozhang Wang added a comment -

          When I run ./gradlew -Dtest.single=RequestResponseSerializationTest core:test, I get:

          -------------------------
          The TaskContainer.add() method has been deprecated and is scheduled to be removed in Gradle 2.0. Please use the create() method instead.
          Building project 'core' with Scala version 2.8.0
          Building project 'perf' with Scala version 2.8.0
          :core:compileJava UP-TO-DATE
          :core:compileScala UP-TO-DATE
          :core:processResources UP-TO-DATE
          :core:classes UP-TO-DATE
          :core:compileTestJava UP-TO-DATE
          :core:compileTestScala UP-TO-DATE
          :core:processTestResources UP-TO-DATE
          :core:testClasses UP-TO-DATE
          :core:test

          BUILD SUCCESSFUL

          Total time: 1 mins 19.067 secs

          --------------

          Which is similar when I run ./gradlew build, except for more tasks UP-TO-DATE, so is there a way to show which tests have been actually executed?

          Show
          Guozhang Wang added a comment - When I run ./gradlew -Dtest.single=RequestResponseSerializationTest core:test, I get: ------------------------- The TaskContainer.add() method has been deprecated and is scheduled to be removed in Gradle 2.0. Please use the create() method instead. Building project 'core' with Scala version 2.8.0 Building project 'perf' with Scala version 2.8.0 :core:compileJava UP-TO-DATE :core:compileScala UP-TO-DATE :core:processResources UP-TO-DATE :core:classes UP-TO-DATE :core:compileTestJava UP-TO-DATE :core:compileTestScala UP-TO-DATE :core:processTestResources UP-TO-DATE :core:testClasses UP-TO-DATE :core:test BUILD SUCCESSFUL Total time: 1 mins 19.067 secs -------------- Which is similar when I run ./gradlew build, except for more tasks UP-TO-DATE, so is there a way to show which tests have been actually executed?
          Hide
          Jakob Homan added a comment -

          There's the test output html, but you can also have gradle be more chatty about which tests it's running/run: http://forums.gradle.org/gradle/topics/whats_new_in_gradle_1_1_test_logging

          Show
          Jakob Homan added a comment - There's the test output html, but you can also have gradle be more chatty about which tests it's running/run: http://forums.gradle.org/gradle/topics/whats_new_in_gradle_1_1_test_logging
          Hide
          Neha Narkhede added a comment -

          Tested patch 14. Here are some review comments -

          uploadCoreArchives_all (mentioned in the README) fails with the following error -
          [nnarkhed@nnarkhed-ld1 kafka]$ ./gradlew uploadCoreArchives_all
          The TaskContainer.add() method has been deprecated and is scheduled to be removed in Gradle 2.0. Please use the create() method instead.
          Building project 'core' with Scala version 2.8.0
          Building project 'perf' with Scala version 2.8.0
          FAILURE: Could not determine which tasks to execute.

          • What went wrong:
            Task 'uploadCoreArchives_all' not found in root project 'kafka'.
          • Try:
            Run gradlew tasks to get a list of available tasks.
            BUILD FAILED
            Total time: 4.577 secs

          :core:uploadCoreArhives fails as well - http://pastebin.com/0NkShchU
          Also, can we fix the comment and documentation here? README should be self sufficient and a lookup of build.gradle should not be required to run commands
          ./gradlew :core:uploadArchives (to test locally, see the comment in build.gradle)

          build_all also failed
          [nnarkhed@nnarkhed-ld1 kafka]$ ./gradlew build_all
          The TaskContainer.add() method has been deprecated and is scheduled to be removed in Gradle 2.0. Please use the create() method instead.
          Building project 'core' with Scala version 2.8.0
          Building project 'perf' with Scala version 2.8.0
          FAILURE: Could not determine which tasks to execute.

          • What went wrong:
            Task 'build_all' not found in root project 'kafka'.
          • Try:
            Run gradlew tasks to get a list of available tasks.
            BUILD FAILED
            Total time: 4.672 secs
          Show
          Neha Narkhede added a comment - Tested patch 14. Here are some review comments - uploadCoreArchives_all (mentioned in the README) fails with the following error - [nnarkhed@nnarkhed-ld1 kafka] $ ./gradlew uploadCoreArchives_all The TaskContainer.add() method has been deprecated and is scheduled to be removed in Gradle 2.0. Please use the create() method instead. Building project 'core' with Scala version 2.8.0 Building project 'perf' with Scala version 2.8.0 FAILURE: Could not determine which tasks to execute. What went wrong: Task 'uploadCoreArchives_all' not found in root project 'kafka'. Try: Run gradlew tasks to get a list of available tasks. BUILD FAILED Total time: 4.577 secs :core:uploadCoreArhives fails as well - http://pastebin.com/0NkShchU Also, can we fix the comment and documentation here? README should be self sufficient and a lookup of build.gradle should not be required to run commands ./gradlew :core:uploadArchives (to test locally, see the comment in build.gradle) build_all also failed [nnarkhed@nnarkhed-ld1 kafka] $ ./gradlew build_all The TaskContainer.add() method has been deprecated and is scheduled to be removed in Gradle 2.0. Please use the create() method instead. Building project 'core' with Scala version 2.8.0 Building project 'perf' with Scala version 2.8.0 FAILURE: Could not determine which tasks to execute. What went wrong: Task 'build_all' not found in root project 'kafka'. Try: Run gradlew tasks to get a list of available tasks. BUILD FAILED Total time: 4.672 secs
          Hide
          Jun Rao added a comment -

          These commands no longer exist in the latest patch. The new commands are in README.

          ./gradlew uploadCoreArchives_all => ./gradlew uploadArchives_all
          ./gradlew build_all => ./gradlew test_all

          Show
          Jun Rao added a comment - These commands no longer exist in the latest patch. The new commands are in README. ./gradlew uploadCoreArchives_all => ./gradlew uploadArchives_all ./gradlew build_all => ./gradlew test_all
          Hide
          Neha Narkhede added a comment - - edited

          Few more comments -

          1. Is it gradle naming convention to have "_" in the task names. For example, test_all instead of just testall?
          2. It is worth trying Jakob's suggestion above as that will improve readability of tests
          3. uploadArchives fails probably because the maven url doesn't exist. But the README should have instructions on how to test it
          Uploading: org/apache/kafka/kafka_2.10.1/0.8.1/kafka_2.10.1-0.8.1.jar to repository remote at http://your.maven.repository
          Error transferring file
          :kafka:core:uploadArchives FAILED
          :uploadCoreArchives_2_10_1 FAILED
          FAILURE: Build failed with an exception.

          • What went wrong:
            Execution failed for task ':core:uploadArchives'.
            > Could not publish configuration 'archives'
            > Error deploying artifact 'org.apache.kafka:kafka_2.10.1:jar': Error deploying artifact: Error transferring file

          4. ./gradlew test_all has some test failures -
          kafka.producer.ProducerTest > testUpdateBrokerPartitionInfo FAILED
          junit.framework.AssertionFailedError at ProducerTest.scala:91
          > Building > :test_core_2_10_1 > :kafka:core:test > 177 tests completed, 1 failed
          8 tests completed, 1 failed
          224 tests completed, 1 failed
          :kafka:core:test FAILED
          :test_core_2_10_1 FAILED
          FAILURE: Build failed with an exception.

          Show
          Neha Narkhede added a comment - - edited Few more comments - 1. Is it gradle naming convention to have "_" in the task names. For example, test_all instead of just testall? 2. It is worth trying Jakob's suggestion above as that will improve readability of tests 3. uploadArchives fails probably because the maven url doesn't exist. But the README should have instructions on how to test it Uploading: org/apache/kafka/kafka_2.10.1/0.8.1/kafka_2.10.1-0.8.1.jar to repository remote at http://your.maven.repository Error transferring file :kafka:core:uploadArchives FAILED :uploadCoreArchives_2_10_1 FAILED FAILURE: Build failed with an exception. What went wrong: Execution failed for task ':core:uploadArchives'. > Could not publish configuration 'archives' > Error deploying artifact 'org.apache.kafka:kafka_2.10.1:jar': Error deploying artifact: Error transferring file 4. ./gradlew test_all has some test failures - kafka.producer.ProducerTest > testUpdateBrokerPartitionInfo FAILED junit.framework.AssertionFailedError at ProducerTest.scala:91 > Building > :test_core_2_10_1 > :kafka:core:test > 177 tests completed, 1 failed 8 tests completed, 1 failed 224 tests completed, 1 failed :kafka:core:test FAILED :test_core_2_10_1 FAILED FAILURE: Build failed with an exception. What went wrong: Execution failed for task ':core:test'. > There were failing tests. See the report at: file:///home/nnarkhed/Projects/kafka/core/build/reports/tests/index.html
          Hide
          Joel Koshy added a comment -

          1 - build.gradle: there should be a programmatic way to enumerate the versions
          for dependsOn as opposed to explicitly enumerating. Ideally, the versions
          should only be specified once in some global list.

          2 - Would love it if we can avoid the additional copyDependantLibs step which
          is extremely annoying. It seems possible by specifying a task dependency
          and it is worth doing.

          3 - gradlew tasks shows a number of tasks. I'm not sure if some are to be used
          or not. E.g., ./gradlew builds, but then is it redundant to jar? Can
          redundant or irrelevant tasks be hidden? Basically it would be nice to
          have consistency between the readme and the tasks output. I'd rather have
          one way to do something than several. I don't care too much but it is a
          bit annoying if the menu shows a lot of options when only half of them are
          to be used.
          4 - kafka-run-class.sh: Why is it there are scala version-specific jars for
          dependant libs and not for kafka-perf* and kafka*? E.g., if we were to do
          jar_all and then kafka-run-class.sh there would be multiple jars on the
          classpath which is a problem - I got an exception.
          5 - Can we get the stand-alone tests available in a test jar or something like
          that? This seems broken on the sbt side as well.
          6 - Would be good to add a line to README.md on how to run a specific unit
          test.
          7 - license plugin: seems very useful. I just ran ./gradlew licenseFormatMain
          and licenseFormatTest and it seems a number of files need license headers.

          Show
          Joel Koshy added a comment - 1 - build.gradle: there should be a programmatic way to enumerate the versions for dependsOn as opposed to explicitly enumerating. Ideally, the versions should only be specified once in some global list. 2 - Would love it if we can avoid the additional copyDependantLibs step which is extremely annoying. It seems possible by specifying a task dependency and it is worth doing. 3 - gradlew tasks shows a number of tasks. I'm not sure if some are to be used or not. E.g., ./gradlew builds, but then is it redundant to jar? Can redundant or irrelevant tasks be hidden? Basically it would be nice to have consistency between the readme and the tasks output. I'd rather have one way to do something than several. I don't care too much but it is a bit annoying if the menu shows a lot of options when only half of them are to be used. 4 - kafka-run-class.sh: Why is it there are scala version-specific jars for dependant libs and not for kafka-perf* and kafka*? E.g., if we were to do jar_all and then kafka-run-class.sh there would be multiple jars on the classpath which is a problem - I got an exception. 5 - Can we get the stand-alone tests available in a test jar or something like that? This seems broken on the sbt side as well. 6 - Would be good to add a line to README.md on how to run a specific unit test. 7 - license plugin: seems very useful. I just ran ./gradlew licenseFormatMain and licenseFormatTest and it seems a number of files need license headers.
          Hide
          Jun Rao added a comment -

          Attach patch v15. Fixed a few more things.

          1. Added tasks to build test jars.
          2. Changed kafka-run-class.sh to pick up the right version of kafka and kafka-perf jar based on the scala version. Also included the jar from all projects (examples, clients, etc) in the script.
          3. Changed unit test reporting. Now the test task prints out the name of each of the tests. If there is a failure, it prints out a longer version of the stack trace.
          4. Figured out a way to force re-running a unit test w/o code change. Documented in README.
          5. Changed tasks from *_all to *All.
          6. Documented all changes in README.

          Also,
          7. ./gradlew build is equivalent to ./gradlew test. I just documented the latter since it's more intuitive.
          8. Those tests failures from 2.10.1 seem to be transient. I ran those a couple of times in both mac and linux and they all pass.

          Show
          Jun Rao added a comment - Attach patch v15. Fixed a few more things. 1. Added tasks to build test jars. 2. Changed kafka-run-class.sh to pick up the right version of kafka and kafka-perf jar based on the scala version. Also included the jar from all projects (examples, clients, etc) in the script. 3. Changed unit test reporting. Now the test task prints out the name of each of the tests. If there is a failure, it prints out a longer version of the stack trace. 4. Figured out a way to force re-running a unit test w/o code change. Documented in README. 5. Changed tasks from *_all to *All. 6. Documented all changes in README. Also, 7. ./gradlew build is equivalent to ./gradlew test. I just documented the latter since it's more intuitive. 8. Those tests failures from 2.10.1 seem to be transient. I ran those a couple of times in both mac and linux and they all pass.
          Hide
          David Arthur added a comment -

          I'm wondering if this isn't done enough to merge. The main requirement here is that developers can build and test using Gradle, and we can do a release. I think both cases are mostly covered.

          Smaller fixes (of which there are plenty) to the project/build can be done outside of this issue. I just worry about this dragging on too long.

          Show
          David Arthur added a comment - I'm wondering if this isn't done enough to merge. The main requirement here is that developers can build and test using Gradle, and we can do a release. I think both cases are mostly covered. Smaller fixes (of which there are plenty) to the project/build can be done outside of this issue. I just worry about this dragging on too long.
          Hide
          Neha Narkhede added a comment -

          +1. I'm also in favor of proceeding with checkin and filing bugs for minor issues to fix right after.

          Show
          Neha Narkhede added a comment - +1. I'm also in favor of proceeding with checkin and filing bugs for minor issues to fix right after.
          Hide
          Joe Stein added a comment -

          I tried to apply v15 (off a fresh trunk clone)

          git am --signoff < kafka-1171_v15.patch

          and got the "Patch does not have a valid e-mail address." error so I tried

          patch -p1 < kafka-1171_v15.patch

          and got all sorts of other errors from that... ./gradlew uploadArchivesAll
          Error: Could not find or load main class org.gradle.wrapper.GradleWrapperMain

          probably doing something wrong let me know so I can try out the upload artifacts, releases, build and the rest...

          didn't have more time to dig into I gotta run out will be back later tonight/tomorrow if anyone can help out thanks!

          Show
          Joe Stein added a comment - I tried to apply v15 (off a fresh trunk clone) git am --signoff < kafka-1171_v15.patch and got the "Patch does not have a valid e-mail address." error so I tried patch -p1 < kafka-1171_v15.patch and got all sorts of other errors from that... ./gradlew uploadArchivesAll Error: Could not find or load main class org.gradle.wrapper.GradleWrapperMain probably doing something wrong let me know so I can try out the upload artifacts, releases, build and the rest... didn't have more time to dig into I gotta run out will be back later tonight/tomorrow if anyone can help out thanks!
          Hide
          David Arthur added a comment -

          Joe Stein I usually use "git apply" when applying patches. I think git-am is specific to the weird git email stuff

          Show
          David Arthur added a comment - Joe Stein I usually use "git apply" when applying patches. I think git-am is specific to the weird git email stuff
          Hide
          Joe Stein added a comment -

          yup, great, thanks that worked

          Show
          Joe Stein added a comment - yup, great, thanks that worked
          Hide
          Jun Rao added a comment -

          Just so that we don't carry the big patch for too long. I will make a small change in kafka-run-class.sh so that both sbt and gradle will work and then check in the patch. We can then iterate on it and remove sbt when everything is done. I will do that if I don't hear any objection in the next hour or so.

          Show
          Jun Rao added a comment - Just so that we don't carry the big patch for too long. I will make a small change in kafka-run-class.sh so that both sbt and gradle will work and then check in the patch. We can then iterate on it and remove sbt when everything is done. I will do that if I don't hear any objection in the next hour or so.
          Hide
          David Arthur added a comment -

          +1

          Show
          David Arthur added a comment - +1
          Hide
          Neha Narkhede added a comment -

          +1

          Show
          Neha Narkhede added a comment - +1
          Hide
          Joel Koshy added a comment -

          +1

          Show
          Joel Koshy added a comment - +1
          Hide
          Jun Rao added a comment -

          Thanks for the review. Patch v15 is now committed. We can leave this jira open for other changes.

          Show
          Jun Rao added a comment - Thanks for the review. Patch v15 is now committed. We can leave this jira open for other changes.

            People

            • Assignee:
              David Arthur
              Reporter:
              David Arthur
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development