Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: packaging
    • Labels:
    1. KAFKA-717-patches-20130720.tgz
      8 kB
      John Landahl
    2. 0004-Fix-cross-compile-of-tests-update-to-2.10.2-and-set-.patch
      3 kB
      Scott Carey
    3. 0003-add-2.9.3.patch
      0.9 kB
      Scott Carey
    4. 0002-java-conversions-changes.patch
      19 kB
      Scott Carey
    5. 0001-common-changes-for-2.10.patch
      9 kB
      Scott Carey
    6. 0003-add-2.9.3.patch
      0.8 kB
      Scott Carey
    7. 0002-java-conversions-changes.patch
      19 kB
      Scott Carey
    8. 0001-common-changes-for-2.10.patch
      9 kB
      Scott Carey
    9. KAFKA-717-complex.patch
      315 kB
      Scott Carey
    10. KAFKA-717-simple.patch
      25 kB
      Scott Carey
    11. 0001-KAFKA-717-Convert-to-scala-2.10.patch
      68 kB
      Matt Christiansen
    12. kafka_scala_2.10.tar.gz
      12 kB
      Darren Sargent

      Issue Links

        Activity

        Hide
        Viktor Taranenko added a comment -

        Yeah. Seems to be so

        Show
        Viktor Taranenko added a comment - Yeah. Seems to be so
        Hide
        Martin Eigenbrodt added a comment - - edited

        Isn't this duplicated and fixed by https://issues.apache.org/jira/browse/KAFKA-1046 ?

        Show
        Martin Eigenbrodt added a comment - - edited Isn't this duplicated and fixed by https://issues.apache.org/jira/browse/KAFKA-1046 ?
        Hide
        Rob Withers added a comment -

        patch tarball no longer applies cleanly to 0.8

        Robs-MacBook-Pro:kafka reefedjib$ git apply ~/Desktop/kafka\ patches/patches/*
        /Users/reefedjib/Desktop/kafka patches/patches/0004-Scala-2.10-support.patch:99: trailing whitespace.
        unmanagedSourceDirectories in Compile <+= (sourceDirectory in Compile, scalaVersion)

        { (s,v) => /Users/reefedjib/Desktop/kafka patches/patches/0004-Scala-2.10-support.patch:101: trailing whitespace. case v if v.startsWith("2.8.") => "2.8.x" /Users/reefedjib/Desktop/kafka patches/patches/0004-Scala-2.10-support.patch:103: trailing whitespace. }

        ))
        /Users/reefedjib/Desktop/kafka patches/patches/0004-Scala-2.10-support.patch:105: new blank line at EOF.
        +
        /Users/reefedjib/Desktop/kafka patches/patches/0015-Scala-2.10-support.patch:442: trailing whitespace.

        • error: patch failed: bin/kafka-run-class.sh:22
          error: bin/kafka-run-class.sh: patch does not apply

        Show
        Rob Withers added a comment - patch tarball no longer applies cleanly to 0.8 Robs-MacBook-Pro:kafka reefedjib$ git apply ~/Desktop/kafka\ patches/patches/* /Users/reefedjib/Desktop/kafka patches/patches/0004-Scala-2.10-support.patch:99: trailing whitespace. unmanagedSourceDirectories in Compile <+= (sourceDirectory in Compile, scalaVersion) { (s,v) => /Users/reefedjib/Desktop/kafka patches/patches/0004-Scala-2.10-support.patch:101: trailing whitespace. case v if v.startsWith("2.8.") => "2.8.x" /Users/reefedjib/Desktop/kafka patches/patches/0004-Scala-2.10-support.patch:103: trailing whitespace. } )) /Users/reefedjib/Desktop/kafka patches/patches/0004-Scala-2.10-support.patch:105: new blank line at EOF. + /Users/reefedjib/Desktop/kafka patches/patches/0015-Scala-2.10-support.patch:442: trailing whitespace. error: patch failed: bin/kafka-run-class.sh:22 error: bin/kafka-run-class.sh: patch does not apply
        Hide
        Rob Withers added a comment -

        I just realized that this is as it should be. The beta1 is an historical branch and new dev for the 0.8.1 stream is on the 0.8 branch. I have a working build, my first truly working kafka build, on my macbook. Tanks!

        Show
        Rob Withers added a comment - I just realized that this is as it should be. The beta1 is an historical branch and new dev for the 0.8.1 stream is on the 0.8 branch. I have a working build, my first truly working kafka build, on my macbook. Tanks!
        Hide
        stephen samuel added a comment -

        I've forked Kafka and patched 0.8 client code to work with scala 2.10. I've released the subsequent artifacts onto maven central.

        <dependency>
        <groupId>com.sksamuel.kafka</groupId>
        <artifactId>kafka_2.10</artifactId>
        <version>0.8.0-beta1</version>
        </dependency>

        or

        http://search.maven.org/#search%7Cga%7C1%7Ca%3A%22kafka_2.10%22

        Hope this is useful for someone. Any issues let me know.

        Show
        stephen samuel added a comment - I've forked Kafka and patched 0.8 client code to work with scala 2.10. I've released the subsequent artifacts onto maven central. <dependency> <groupId>com.sksamuel.kafka</groupId> <artifactId>kafka_2.10</artifactId> <version>0.8.0-beta1</version> </dependency> or http://search.maven.org/#search%7Cga%7C1%7Ca%3A%22kafka_2.10%22 Hope this is useful for someone. Any issues let me know.
        Hide
        Darren Sargent added a comment -

        I'll be out of the office 7/27 through 8/4, checking email from time to time. If your request is urgent, please contact anyone on the Platform team.

        Show
        Darren Sargent added a comment - I'll be out of the office 7/27 through 8/4, checking email from time to time. If your request is urgent, please contact anyone on the Platform team.
        Hide
        Rob Withers added a comment -

        we tried the tgz of 22 patches against the beta1-candidate branch and it failed. We got it working with the 0.8.0 branch. Is there a possibility someone could provide a tgz of patches that applies to the beta1-candidate branch? Thanks, much.

        Show
        Rob Withers added a comment - we tried the tgz of 22 patches against the beta1-candidate branch and it failed. We got it working with the 0.8.0 branch. Is there a possibility someone could provide a tgz of patches that applies to the beta1-candidate branch? Thanks, much.
        Hide
        John Landahl added a comment - - edited

        The June 14 patch set no longer applies cleanly. I've re-applied the patches to the latest 0.8 branch and attached a new set as KAFKA-717-patches-20130720.tgz. There is a separate patch per file for better shelf life.

        To apply: git checkout 0.8; git apply *.patch

        Show
        John Landahl added a comment - - edited The June 14 patch set no longer applies cleanly. I've re-applied the patches to the latest 0.8 branch and attached a new set as KAFKA-717 -patches-20130720.tgz. There is a separate patch per file for better shelf life. To apply: git checkout 0.8; git apply *.patch
        Hide
        Scott Carey added a comment -

        The latter plan looks good to me.

        Show
        Scott Carey added a comment - The latter plan looks good to me.
        Hide
        Jay Kreps added a comment -

        Yeah I think we all want to do this, and likely it will work.

        LinkedIn uses 2.8.0 and we can't upgrade our clients until all consumers are upgraded. This is happening now but takes a while (we have hundreds of applications) so we aren't going to be off 2.8.0 in the next few weeks for 0.8.

        LinkedIn can do one of two things:

        • Work off a local fork for linkedin temporarily for 0.8 so we can keep running 2.8
        • Wait until 0.8.1 and let people who really want 2.10 apply the patch

        Personally I think the project is better off with the later strategy so that 0.8 is getting any fixes from our production deploys, but you might disagree.

        Show
        Jay Kreps added a comment - Yeah I think we all want to do this, and likely it will work. LinkedIn uses 2.8.0 and we can't upgrade our clients until all consumers are upgraded. This is happening now but takes a while (we have hundreds of applications) so we aren't going to be off 2.8.0 in the next few weeks for 0.8. LinkedIn can do one of two things: Work off a local fork for linkedin temporarily for 0.8 so we can keep running 2.8 Wait until 0.8.1 and let people who really want 2.10 apply the patch Personally I think the project is better off with the later strategy so that 0.8 is getting any fixes from our production deploys, but you might disagree.
        Hide
        Scott Carey added a comment -

        that would be easiest for us would be post-0.8 (i.e. 0.8.1/trunk)

        (Can the wiki-text plugin be enabled for this project?)

        That's fine, I'm happy maintaining our own fork to handle 2.10.x for now. This is about how to move forward long term, which requires trimming support for old versions. 2.11 (a 'performance' release) will be out in 6 months or so, and 2.8.x will become even more difficult to support in conjunction.

        In this case, the big pain point is the Java conversions/converters code in the Java API. New client APIs in Kafka can simultaneously make the code more portable across Scala versions, since incompatibilities are generally due to library changes and not language changes.

        Show
        Scott Carey added a comment - that would be easiest for us would be post-0.8 (i.e. 0.8.1/trunk) (Can the wiki-text plugin be enabled for this project?) That's fine, I'm happy maintaining our own fork to handle 2.10.x for now. This is about how to move forward long term, which requires trimming support for old versions. 2.11 (a 'performance' release) will be out in 6 months or so, and 2.8.x will become even more difficult to support in conjunction. In this case, the big pain point is the Java conversions/converters code in the Java API. New client APIs in Kafka can simultaneously make the code more portable across Scala versions, since incompatibilities are generally due to library changes and not language changes.
        Hide
        Scott Carey added a comment -

        As it stands now folks are already running 0.8.0-beta1 in production

        We have been up and running in production with 0.8 before beta-1, with Scala 2.10.1. About a month ago we moved to the most recent 0.8 with Scala 2.10.2. 2.10.x seems to provide a minor performance boost as well (faster Scala collections).
        Our load is not as high as LI and some others, but it is 250GB /day (after compression), involves multiple partition topics, and heavily leverages the mirroring. The performance was worse than 0.7.x at first, but it is catching back up.

        Compiling with 2.10.x actually highlights a LOT of warnings that could indicate bugs in Kafka that 2.8.x compiler versions do not display. I am more worried about those, possible bugs, than what upgrading to a newer Scala version brings.

        I am much more concerned with the need for cross-compile for clients than for the broker. The broker is isolated and can be tested on its own and default to a newer Scala version. Consumer or Producer client libraries however have to live in another application, and that application may have other dependencies that constrain what version is appropriate.

        Show
        Scott Carey added a comment - As it stands now folks are already running 0.8.0-beta1 in production We have been up and running in production with 0.8 before beta-1, with Scala 2.10.1. About a month ago we moved to the most recent 0.8 with Scala 2.10.2. 2.10.x seems to provide a minor performance boost as well (faster Scala collections). Our load is not as high as LI and some others, but it is 250GB /day (after compression), involves multiple partition topics, and heavily leverages the mirroring. The performance was worse than 0.7.x at first, but it is catching back up. Compiling with 2.10.x actually highlights a LOT of warnings that could indicate bugs in Kafka that 2.8.x compiler versions do not display. I am more worried about those, possible bugs, than what upgrading to a newer Scala version brings. I am much more concerned with the need for cross-compile for clients than for the broker. The broker is isolated and can be tested on its own and default to a newer Scala version. Consumer or Producer client libraries however have to live in another application, and that application may have other dependencies that constrain what version is appropriate.
        Hide
        Jay Kreps added a comment -

        We are up for switching over at LinkedIn, and will let people know how it goes. We have actually been delaying to get the 0.8 release done in hope of changing one thing at a time. The timeframe that would be easiest for us would be post-0.8 (i.e. 0.8.1/trunk).

        Show
        Jay Kreps added a comment - We are up for switching over at LinkedIn, and will let people know how it goes. We have actually been delaying to get the 0.8 release done in hope of changing one thing at a time. The timeframe that would be easiest for us would be post-0.8 (i.e. 0.8.1/trunk).
        Hide
        Joe Stein added a comment -

        Moving away from 2.8.X for the broker build would be more feasible when folks running brokers in production are running them on 2.9.X or even 2.10.X and right now I don't know of anyone running a broker not on 2.8.X with any real volume (but that does not mean its not happening just I don't know about it). I will work over the next few weeks to start to pull more of this information together so its empirical and not assumed.

        As it stands now folks are already running 0.8.0-beta1 in production and there is only a certain amount of beta that production operation folks will tolerate and upgrading from Scala 2.8.0 to another version for companies that have terabytes a day going through Kafka as a key part of their operations may not be feasible just yet.

        While I agree with you 100% I also understand and appreciate how we need to support this project (lots of folks to support in different ways) and should be supporting this project for the stability it offers and rushing ahead has to be tempered with production support by the Apache Kafka community and PMC.

        From my perspective the scalaVersion default is for the broker and the cross compiles are for different needs of producer/consumer (at least how I see it) and now that we are publishing to maven the default should matter even more only for the broker (again IMHO)

        That all being said I looked through and get the patches now and will give these a shot in order using the 06/14/2013 dates

        0001-common-changes-for-2.10.patch
        0002-java-conversions-changes.patch
        0003-add-2.9.3.patch
        0004-Fix-cross-compile-of-tests-update-to-2.10.2-and-set-.patch

        I also have some ideas about ways to making this available sooner for some things I am working on the next few weeks moving forward.

        Thank you for the patches and patience. The best is yet to come!

        Show
        Joe Stein added a comment - Moving away from 2.8.X for the broker build would be more feasible when folks running brokers in production are running them on 2.9.X or even 2.10.X and right now I don't know of anyone running a broker not on 2.8.X with any real volume (but that does not mean its not happening just I don't know about it). I will work over the next few weeks to start to pull more of this information together so its empirical and not assumed. As it stands now folks are already running 0.8.0-beta1 in production and there is only a certain amount of beta that production operation folks will tolerate and upgrading from Scala 2.8.0 to another version for companies that have terabytes a day going through Kafka as a key part of their operations may not be feasible just yet. While I agree with you 100% I also understand and appreciate how we need to support this project (lots of folks to support in different ways) and should be supporting this project for the stability it offers and rushing ahead has to be tempered with production support by the Apache Kafka community and PMC. From my perspective the scalaVersion default is for the broker and the cross compiles are for different needs of producer/consumer (at least how I see it) and now that we are publishing to maven the default should matter even more only for the broker (again IMHO) That all being said I looked through and get the patches now and will give these a shot in order using the 06/14/2013 dates 0001-common-changes-for-2.10.patch 0002-java-conversions-changes.patch 0003-add-2.9.3.patch 0004-Fix-cross-compile-of-tests-update-to-2.10.2-and-set-.patch I also have some ideas about ways to making this available sooner for some things I am working on the next few weeks moving forward. Thank you for the patches and patience. The best is yet to come!
        Hide
        Scott Carey added a comment -

        looking at the simple patch not sure changing the scalaVersion default from 2.8.0 to 2.8.2 is a great idea and what the reasoning was behind it

        Since 2.8.0 support is dropped, it had to change to something. I'd rather drop 2.8.x entirely. 2.8.x is not source-compatible with 2.10.x in several places.

        More importantly the latest patch series is NOT the 'simple' patch. Look at the dates of the files above, use 'git apply' on the numbered patch series from June 14.

        will see if I can make any headway on trunk with the patches

        I can probably rebase them to trunk. Assuming trunk has had its merge from 0.8 completed, otherwise it will be a mess.

        without some good reasoning for it

        How about that most new Scala libraries these days no longer cross-compile to 2.8.x?

        Show
        Scott Carey added a comment - looking at the simple patch not sure changing the scalaVersion default from 2.8.0 to 2.8.2 is a great idea and what the reasoning was behind it Since 2.8.0 support is dropped, it had to change to something. I'd rather drop 2.8.x entirely. 2.8.x is not source-compatible with 2.10.x in several places. More importantly the latest patch series is NOT the 'simple' patch. Look at the dates of the files above, use 'git apply' on the numbered patch series from June 14. will see if I can make any headway on trunk with the patches I can probably rebase them to trunk. Assuming trunk has had its merge from 0.8 completed, otherwise it will be a mess. without some good reasoning for it How about that most new Scala libraries these days no longer cross-compile to 2.8.x?
        Hide
        Joe Stein added a comment -

        looking at the simple patch not sure changing the scalaVersion default from 2.8.0 to 2.8.2 is a great idea and what the reasoning was behind it

        will see if I can make any headway on trunk with the patches but would not commit the version change if I do get it working without some good reasoning for it. the complex version appears to have a lot of unnecessary changes bleeding into the patch just for 2.10.X support ... will also go through the others for 2.9.3 might have to create one new patch that is clean for this ticket.. well see dunno yet

        Show
        Joe Stein added a comment - looking at the simple patch not sure changing the scalaVersion default from 2.8.0 to 2.8.2 is a great idea and what the reasoning was behind it will see if I can make any headway on trunk with the patches but would not commit the version change if I do get it working without some good reasoning for it. the complex version appears to have a lot of unnecessary changes bleeding into the patch just for 2.10.X support ... will also go through the others for 2.9.3 might have to create one new patch that is clean for this ticket.. well see dunno yet
        Hide
        Matei Zaharia added a comment -

        Sounds good! Glad to hear you guys are looking at it.

        Show
        Matei Zaharia added a comment - Sounds good! Glad to hear you guys are looking at it.
        Hide
        Jay Kreps added a comment -

        Yeah we can do it in 0.8.1 which should happen fairly quickly after 0.8 but we are really trying to lock down 0.8 to critical fixes and this is more of a new feature (though obviously desirable). Sorry for the hassle.

        Show
        Jay Kreps added a comment - Yeah we can do it in 0.8.1 which should happen fairly quickly after 0.8 but we are really trying to lock down 0.8 to critical fixes and this is more of a new feature (though obviously desirable). Sorry for the hassle.
        Hide
        Jun Rao added a comment -

        We should take this patch in trunk, instead of 0.8.

        Show
        Jun Rao added a comment - We should take this patch in trunk, instead of 0.8.
        Hide
        Matei Zaharia added a comment -

        I'm curious, is this intended to be included in 0.8? I use Kafka in my project (www.spark-project.org, recently entered Apache Incubator) and we've had to ship our own JAR to get Scala 2.9 support, and now we might have to do the same for 2.10 when we update to 2.10.

        Show
        Matei Zaharia added a comment - I'm curious, is this intended to be included in 0.8? I use Kafka in my project (www.spark-project.org, recently entered Apache Incubator) and we've had to ship our own JAR to get Scala 2.9 support, and now we might have to do the same for 2.10 when we update to 2.10.
        Hide
        Sriram Subramanian added a comment -

        I get the following errors when I try to apply the patches to 0.8 in the order that were added on 14th Jun -

        1. patch 1 failed

        ../Downloads/0001-common-changes-for-2.10.patch:34: trailing whitespace.
        unmanagedSourceDirectories in Compile <+= (sourceDirectory in Compile, scalaVersion)

        { (s,v) => ../Downloads/0001-common-changes-for-2.10.patch:36: trailing whitespace. case v if v.startsWith("2.8.") => "2.8.x" ../Downloads/0001-common-changes-for-2.10.patch:38: trailing whitespace. }

        ))
        ../Downloads/0001-common-changes-for-2.10.patch:54: trailing whitespace.

        • ../Downloads/0001-common-changes-for-2.10.patch:69: trailing whitespace.

        • Indicates that the annotated class is meant to be threadsafe. For an abstract class it is an part of the interface that an implementation
          error: patch failed: project/Build.scala:28
          error: project/Build.scala: patch does not apply

        2. patch 3 failed

        error: patch failed: project/Build.scala:28
        error: project/Build.scala: patch does not apply

        3. patch 4 failed

        error: patch failed: core/build.sbt:22
        error: core/build.sbt: patch does not apply
        error: patch failed: core/src/test/scala/unit/kafka/metrics/KafkaTimerTest.scala:36
        error: core/src/test/scala/unit/kafka/metrics/KafkaTimerTest.scala: patch does not apply
        error: patch failed: project/Build.scala:28
        error: project/Build.scala: patch does not apply

        I used git apply

        Show
        Sriram Subramanian added a comment - I get the following errors when I try to apply the patches to 0.8 in the order that were added on 14th Jun - 1. patch 1 failed ../Downloads/0001-common-changes-for-2.10.patch:34: trailing whitespace. unmanagedSourceDirectories in Compile <+= (sourceDirectory in Compile, scalaVersion) { (s,v) => ../Downloads/0001-common-changes-for-2.10.patch:36: trailing whitespace. case v if v.startsWith("2.8.") => "2.8.x" ../Downloads/0001-common-changes-for-2.10.patch:38: trailing whitespace. } )) ../Downloads/0001-common-changes-for-2.10.patch:54: trailing whitespace. ../Downloads/0001-common-changes-for-2.10.patch:69: trailing whitespace. Indicates that the annotated class is meant to be threadsafe. For an abstract class it is an part of the interface that an implementation error: patch failed: project/Build.scala:28 error: project/Build.scala: patch does not apply 2. patch 3 failed error: patch failed: project/Build.scala:28 error: project/Build.scala: patch does not apply 3. patch 4 failed error: patch failed: core/build.sbt:22 error: core/build.sbt: patch does not apply error: patch failed: core/src/test/scala/unit/kafka/metrics/KafkaTimerTest.scala:36 error: core/src/test/scala/unit/kafka/metrics/KafkaTimerTest.scala: patch does not apply error: patch failed: project/Build.scala:28 error: project/Build.scala: patch does not apply I used git apply
        Hide
        Martin Eigenbrodt added a comment -

        +1
        Maybe usefull for someone: I applied the patches to 0.8beta1 and put it on github: https://github.com/martinei/kafka/tree/0.8.0-beta1-candidate1-scala2.10

        Show
        Martin Eigenbrodt added a comment - +1 Maybe usefull for someone: I applied the patches to 0.8beta1 and put it on github: https://github.com/martinei/kafka/tree/0.8.0-beta1-candidate1-scala2.10
        Hide
        Matt Christiansen added a comment -

        +1 for the patch too

        I would love to see this committed as currently I have to apply it to our own build of 0.8 would be great to stop doing that.

        Show
        Matt Christiansen added a comment - +1 for the patch too I would love to see this committed as currently I have to apply it to our own build of 0.8 would be great to stop doing that.
        Hide
        Jan Prach added a comment -

        +1 for Scala 2.10 support

        The patches are obviously correct (mostly just JavaConverters stuff). Let's commit it!

        Thank you for maintaining the patches, Scott!

        Show
        Jan Prach added a comment - +1 for Scala 2.10 support The patches are obviously correct (mostly just JavaConverters stuff). Let's commit it! Thank you for maintaining the patches, Scott!
        Hide
        Scott Carey added a comment -

        Attached are patches for updating Scala version support. 2.8.0 is dropped, 2.9.3 and 2.10.2 are. The default is set to 2.10.2.

        This applies to the current 0.8 (23acbd309f5e17de71db46cb6f1a60c8d38ea4e4) branch and 0.8beta1.

        The patch series is best applied with 'git apply' as before.

        Show
        Scott Carey added a comment - Attached are patches for updating Scala version support. 2.8.0 is dropped, 2.9.3 and 2.10.2 are. The default is set to 2.10.2. This applies to the current 0.8 (23acbd309f5e17de71db46cb6f1a60c8d38ea4e4) branch and 0.8beta1. The patch series is best applied with 'git apply' as before.
        Hide
        Richard Grossman added a comment -

        Good to know
        Thanks but I've found a fork on Github that already made the job done
        Thank you

        Show
        Richard Grossman added a comment - Good to know Thanks but I've found a fork on Github that already made the job done Thank you
        Hide
        Scott Carey added a comment -

        This patch series applies to 0.8.

        The patch contains new files and directories, you may have better luck with 'git apply' than with 'patch'. The patch was formatted with 'git-format-patch' so that 'git apply' should handle everything for you.

        http://eothred.wordpress.com/2011/07/02/git-and-patches/

        Show
        Scott Carey added a comment - This patch series applies to 0.8. The patch contains new files and directories, you may have better luck with 'git apply' than with 'patch'. The patch was formatted with 'git-format-patch' so that 'git apply' should handle everything for you. http://eothred.wordpress.com/2011/07/02/git-and-patches/
        Hide
        Scott Carey added a comment -

        The patch was meant to apply to the 0.8 branch. The directories you list are new. If you do not need Scala 2.8.0 support I suggest the simple variation.

        I will post the simple variation patch again after rebasing to the current 0.8 head.

        Show
        Scott Carey added a comment - The patch was meant to apply to the 0.8 branch. The directories you list are new. If you do not need Scala 2.8.0 support I suggest the simple variation. I will post the simple variation patch again after rebasing to the current 0.8 head.
        Hide
        Richard Grossman added a comment - - edited

        Hi

        I've some problem with how to apply the patch, I need to compile kafka to run with scala 2.10
        I've git cloned the repository

        then applied the patch KAFKA-717-simple.patch
        then I try to package ./sbt "++2.10.1 package"
        But I still got errors likes because some file seems not to be patched

        C:\sources\kafka\core\src\main\scala\kafka\server\TopicConfigManager.scala:80: value asBuffer is not a member of object scala.collection.JavaConversions
        [error] processConfigChanges(JavaConversions.asBuffer(configChanges).sorted)
        [error] ^

        making me thinking the patch as not been applied as expected
        please help
        thanks

        Show
        Richard Grossman added a comment - - edited Hi I've some problem with how to apply the patch, I need to compile kafka to run with scala 2.10 I've git cloned the repository then applied the patch KAFKA-717 -simple.patch then I try to package ./sbt "++2.10.1 package" But I still got errors likes because some file seems not to be patched C:\sources\kafka\core\src\main\scala\kafka\server\TopicConfigManager.scala:80: value asBuffer is not a member of object scala.collection.JavaConversions [error] processConfigChanges(JavaConversions.asBuffer(configChanges).sorted) [error] ^ making me thinking the patch as not been applied as expected please help thanks
        Hide
        Scott Carey added a comment -

        The patches here implement conditional compilation of some sources based on the version.

        core/src/main/scala-2.8.x
        and
        core/src/main/scala-2.9.x
        have been added, to the normal
        core/src/main/scala
        (and similar for tests)

        2.8.x builds add the former, and 2.9.x and 2.10.x builds use the latter.

        There are two patches,
        KAFKA-717-simple.patch – this variation only requires one conditionally compiled file, but does not support 2.8.0 – it supports 2.8.2, 2.9.1, 2.9.2, 2.9.3 and 2.10.1

        KAFKA-717-complex.patch supports all the versions above as well as 2.8.0, but it needs to conditionally compile 15 more files, some of them large.

        Simply from the size of the patch file, one can see how much more difficult it is to support 2.8.0 cross compilation with 2.10.0 than 2.8.2. The 'complex' patch is 12x as large.

        Show
        Scott Carey added a comment - The patches here implement conditional compilation of some sources based on the version. core/src/main/scala-2.8.x and core/src/main/scala-2.9.x have been added, to the normal core/src/main/scala (and similar for tests) 2.8.x builds add the former, and 2.9.x and 2.10.x builds use the latter. There are two patches, KAFKA-717 -simple.patch – this variation only requires one conditionally compiled file, but does not support 2.8.0 – it supports 2.8.2, 2.9.1, 2.9.2, 2.9.3 and 2.10.1 KAFKA-717 -complex.patch supports all the versions above as well as 2.8.0, but it needs to conditionally compile 15 more files, some of them large. Simply from the size of the patch file, one can see how much more difficult it is to support 2.8.0 cross compilation with 2.10.0 than 2.8.2. The 'complex' patch is 12x as large.
        Hide
        Scott Carey added a comment -

        To support all versions concurrently, we need a patch that moves the affected files to special directories per version.

        This s.o. question seems to have the most information I can find regarding the issue and how to solve it: http://stackoverflow.com/questions/13872226/how-to-support-multiple-scala-versions-in-a-library

        Show
        Scott Carey added a comment - To support all versions concurrently, we need a patch that moves the affected files to special directories per version. This s.o. question seems to have the most information I can find regarding the issue and how to solve it: http://stackoverflow.com/questions/13872226/how-to-support-multiple-scala-versions-in-a-library
        Hide
        Matt Christiansen added a comment -

        Here is an updated patch built from latest HEAD on the 0.8 branch. I also bumbed it to scala 2.10.1 which has some minor bug fixes over 2.10.0. All unit tests pass locally and I did a basic test using the Console producer /consumer.

        The patch that Darren provided I have tested more extensively but no longer applied to HEAD.

        Show
        Matt Christiansen added a comment - Here is an updated patch built from latest HEAD on the 0.8 branch. I also bumbed it to scala 2.10.1 which has some minor bug fixes over 2.10.0. All unit tests pass locally and I did a basic test using the Console producer /consumer. The patch that Darren provided I have tested more extensively but no longer applied to HEAD.
        Hide
        Darren Sargent added a comment -

        Scott is correct - the patch was against the head of the 0.8 branch on March 4th (see history).

        I have retrofitted the changes to the 0.8 head as of a couple days ago – I can upload a newer patch if it's useful. Some code changes are required for Scala 2.10 compatibility.

        Show
        Darren Sargent added a comment - Scott is correct - the patch was against the head of the 0.8 branch on March 4th (see history). I have retrofitted the changes to the 0.8 head as of a couple days ago – I can upload a newer patch if it's useful. Some code changes are required for Scala 2.10 compatibility.
        Hide
        Scott Carey added a comment - - edited

        The patch is already attached to this ticket, it is inside the tar.gz file.

        In order to support 2.10, 2.8.x support needs to be dropped or a file or two will have to differ between the versions. See my notes from March 4 above.

        Edit: I believe the patch attached was for trunk at about March 4.

        Show
        Scott Carey added a comment - - edited The patch is already attached to this ticket, it is inside the tar.gz file. In order to support 2.10, 2.8.x support needs to be dropped or a file or two will have to differ between the versions. See my notes from March 4 above. Edit: I believe the patch attached was for trunk at about March 4.
        Hide
        Neha Narkhede added a comment -

        Viktor, few questions -

        1. Will there be any code changes required to support building against Scala 2.10 ?
        2. Would you mind submitting a patch to give us an idea of what those changes look like ?

        Show
        Neha Narkhede added a comment - Viktor, few questions - 1. Will there be any code changes required to support building against Scala 2.10 ? 2. Would you mind submitting a patch to give us an idea of what those changes look like ?
        Hide
        Viktor Taranenko added a comment -

        Is it possible to get this issue done for 0.8 release?

        Show
        Viktor Taranenko added a comment - Is it possible to get this issue done for 0.8 release?
        Hide
        Darren Sargent added a comment -

        Fixes to allow Kafka to build against Scala 2.10.

        Show
        Darren Sargent added a comment - Fixes to allow Kafka to build against Scala 2.10.
        Hide
        Scott Carey added a comment -

        I take that back, scala.annotation.StaticAnnotation moved from scala.StaticAnnotation, to so you cannot cross-compile 2.8.2 with 2.10 if that is referenced.

        It would appear that you can cross compile 2.9.x with 2.10.x, 2.8.x with 2.9.x, but not 2.8.x with 2.10.x.

        Is the StaticAnnotation required? without it, it appears cross-compilation between 2.8.2 to 2.10.0 is possible.

        Show
        Scott Carey added a comment - I take that back, scala.annotation.StaticAnnotation moved from scala.StaticAnnotation, to so you cannot cross-compile 2.8.2 with 2.10 if that is referenced. It would appear that you can cross compile 2.9.x with 2.10.x, 2.8.x with 2.9.x, but not 2.8.x with 2.10.x. Is the StaticAnnotation required? without it, it appears cross-compilation between 2.8.2 to 2.10.0 is possible.
        Hide
        Scott Carey added a comment -

        It appears that the changes are back-compatible with 2.8.1+ (the removed Scala APIs have been deprecated since then with their replacements). It also moves some use of JavaConversions to JavaConverters, which is highly recommended:
        http://stackoverflow.com/questions/8301947/what-is-the-difference-between-javaconverters-and-javaconversions-in-scala

        They are not compatible with 2.8.0, because it does not have JavaConverters. Are there users of Kafka that require 2.8.0 and can't use 2.8.1?

        Show
        Scott Carey added a comment - It appears that the changes are back-compatible with 2.8.1+ (the removed Scala APIs have been deprecated since then with their replacements). It also moves some use of JavaConversions to JavaConverters, which is highly recommended: http://stackoverflow.com/questions/8301947/what-is-the-difference-between-javaconverters-and-javaconversions-in-scala They are not compatible with 2.8.0, because it does not have JavaConverters. Are there users of Kafka that require 2.8.0 and can't use 2.8.1?
        Hide
        Darren Sargent added a comment -

        I got Kafka 0.8 to compile and run with Scala 2.10.0. I fixed some of the warnings due to bare catch in favor of catch Exception – the behavior for a bare catch would be to catch Throwable, but apparently catch Throwable is evil because internally scala throws Throwables for flow control.

        I only tested this with the basic console producer and consumer so caveat emptor.

        Show
        Darren Sargent added a comment - I got Kafka 0.8 to compile and run with Scala 2.10.0. I fixed some of the warnings due to bare catch in favor of catch Exception – the behavior for a bare catch would be to catch Throwable, but apparently catch Throwable is evil because internally scala throws Throwables for flow control. I only tested this with the basic console producer and consumer so caveat emptor.

          People

          • Assignee:
            Unassigned
            Reporter:
            Viktor Taranenko
          • Votes:
            8 Vote for this issue
            Watchers:
            29 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development