Kafka
  1. Kafka
  2. KAFKA-733

Fat jar option for build, or override for ivy cache location

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: packaging
    • Labels:

      Description

      Need some kind of self-contained mechanism for running kafka to get around the following:
      1) The location of the source checkout/build is not necessarily the same place where it will be running (the build location and that user's ivy cache dir) potentially leading to sync problems (forgetting the ivy dir) or just adding overhead to the deployment process (additional steps to remember introduces more chances for mistakes)
      2) The user running the kafka service in a production setting may not even have a real home directory

      Think something like a 'fat jar' packaging (something that contains all necessary jar versions in one convenient place) would simplify deployment and reduce the chance for error (only one lib package to worry about, and it contains everything needed) and would be a little more production friendly

      1. KAFKA-733.patch
        1 kB
        Dave DeMaagd
      2. KAFKA-733-2.patch
        2 kB
        Dave DeMaagd

        Activity

        Dave DeMaagd created issue -
        Hide
        Dave DeMaagd added a comment -

        Mostly cut and paste patch based on https://github.com/sbt/sbt-assembly - does appear to work.

        Show
        Dave DeMaagd added a comment - Mostly cut and paste patch based on https://github.com/sbt/sbt-assembly - does appear to work.
        Dave DeMaagd made changes -
        Field Original Value New Value
        Attachment KAFKA-733.patch [ 12566370 ]
        Dave DeMaagd made changes -
        Labels patch
        Swapnil Ghike made changes -
        Labels patch bugs patch
        Swapnil Ghike made changes -
        Labels bugs patch bugs
        Swapnil Ghike made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Swapnil Ghike made changes -
        Assignee Dave DeMaagd [ demaagd ]
        Hide
        derek added a comment -

        Patch looks good. Two questions: why was the merge strategy overridden? What's in the patch is basically the default minus a few cases. Also, would it be useful to set a main class for the JAR?

        Show
        derek added a comment - Patch looks good. Two questions: why was the merge strategy overridden? What's in the patch is basically the default minus a few cases. Also, would it be useful to set a main class for the JAR?
        Hide
        Dave DeMaagd added a comment -

        The mergeStrategy was put in place because there was a conflict with zkclient:

        java.lang.RuntimeException: deduplicate: different file contents found in the following:
        /...../.ivy2/cache/com.github.sgroschupf/zkclient/jars/zkclient-0.1.jar:org/I0Itec/zkclient/util/ZkPathUtil.class
        /...../kafka-git/0.8/core/lib/zkclient-20120522.jar:org/I0Itec/zkclient/util/ZkPathUtil.class

        The reason I chose 'MergeStrategy.last' was mostly a coin toss. If there is a better argument (any sound reasoning is likely to trump my coin toss here), I welcome suggestions to make this cover more cases than I had in mind.

        I didn't add a main class to it because the the goal I had in mind was being able to run the tools and admin bits without needing to cart around (and keep track of for upgrades) a large number of files. A single jar file to track is far easier from an operability standpoint for me to manage.

        Show
        Dave DeMaagd added a comment - The mergeStrategy was put in place because there was a conflict with zkclient: java.lang.RuntimeException: deduplicate: different file contents found in the following: /...../.ivy2/cache/com.github.sgroschupf/zkclient/jars/zkclient-0.1.jar:org/I0Itec/zkclient/util/ZkPathUtil.class /...../kafka-git/0.8/core/lib/zkclient-20120522.jar:org/I0Itec/zkclient/util/ZkPathUtil.class The reason I chose 'MergeStrategy.last' was mostly a coin toss. If there is a better argument (any sound reasoning is likely to trump my coin toss here), I welcome suggestions to make this cover more cases than I had in mind. I didn't add a main class to it because the the goal I had in mind was being able to run the tools and admin bits without needing to cart around (and keep track of for upgrades) a large number of files. A single jar file to track is far easier from an operability standpoint for me to manage.
        Hide
        derek added a comment -

        If there are conflicting zookeeper libs one should probably be removed, either by removing the checked-in zkclient-20120522.jar or removing/excluding whatever dep is bringing in zkclient-0.1.jar. I'll take a look at the deps and see if I can sort that out. As for the main class, setting one in the JAR won't preclude you from running other main classes from the same jar, it just sets the Main-Class property in the manifest.

        Show
        derek added a comment - If there are conflicting zookeeper libs one should probably be removed, either by removing the checked-in zkclient-20120522.jar or removing/excluding whatever dep is bringing in zkclient-0.1.jar. I'll take a look at the deps and see if I can sort that out. As for the main class, setting one in the JAR won't preclude you from running other main classes from the same jar, it just sets the Main-Class property in the manifest.
        Hide
        derek added a comment -

        OK, you should either remove the zkclient-20120522.jar from core/lib or remove the dep from core/build.sbt. I can't see any sane reason to have two versions of the same client. I'd vote to remove the internal JAR and go with the build.sbt dep. And then get rid of the merge strategy

        Show
        derek added a comment - OK, you should either remove the zkclient-20120522.jar from core/lib or remove the dep from core/build.sbt. I can't see any sane reason to have two versions of the same client. I'd vote to remove the internal JAR and go with the build.sbt dep. And then get rid of the merge strategy
        Hide
        Joe Stein added a comment - - edited

        I like the idea of assembly but not everyone is going to want/like another dependency so we adding a dependency for running the server without providing another option.

        In either case something has to get done here KAFKA-760 for how to start the server for different build options this would be another option

        Show
        Joe Stein added a comment - - edited I like the idea of assembly but not everyone is going to want/like another dependency so we adding a dependency for running the server without providing another option. In either case something has to get done here KAFKA-760 for how to start the server for different build options this would be another option
        Hide
        Dave DeMaagd added a comment -

        KAFKA-733-2.patch (attached in a moment) removes the 'additional' dependency - com.eed3si9n/sbt-assembly is already there (the second instance is removed in the patch), so in effect all the patch is doing is configuring the existing dep.

        As for the zk client libs - I will have to defer to someone else on the right handling of it (someone with more familiarity with the code level differences between them).

        Show
        Dave DeMaagd added a comment - KAFKA-733 -2.patch (attached in a moment) removes the 'additional' dependency - com.eed3si9n/sbt-assembly is already there (the second instance is removed in the patch), so in effect all the patch is doing is configuring the existing dep. As for the zk client libs - I will have to defer to someone else on the right handling of it (someone with more familiarity with the code level differences between them).
        Dave DeMaagd made changes -
        Attachment KAFKA-733-2.patch [ 12570831 ]
        Hide
        Jun Rao added a comment -

        This is already fixed as part of the sbt enhancement.

        Show
        Jun Rao added a comment - This is already fixed as part of the sbt enhancement.
        Jun Rao made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.8 [ 12317244 ]
        Resolution Duplicate [ 3 ]

          People

          • Assignee:
            Dave DeMaagd
            Reporter:
            Dave DeMaagd
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development