Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.1.0
    • Component/s: None
    • Labels:
      None

      Description

      I was looking into morphline and Apache Flume and noticed that CDK was needed in order to get the morphline jars.

      So let's package morphline so other projects can benefit from it!

      1. 0001-BIGTOP-1149.-Package-CDK.patch
        14 kB
        Sean Mackrory
      2. 0001-BIGTOP-1149.-Package-Kite.patch
        14 kB
        Sean Mackrory
      3. BIGTOP-1149.1.patch
        23 kB
        YoungWoo Kim
      4. BIGTOP-1149.2.patch
        24 kB
        YoungWoo Kim
      5. BIGTOP-1149.3.patch
        24 kB
        YoungWoo Kim
      6. BIGTOP-1149.4.patch
        24 kB
        YoungWoo Kim
      7. BIGTOP-1149.5.patch
        24 kB
        YoungWoo Kim
      8. BIGTOP-1149.6.patch
        25 kB
        YoungWoo Kim
      9. BIGTOP-1149.7.patch
        25 kB
        YoungWoo Kim

        Activity

        Hide
        cos Konstantin Boudnik added a comment -

        what's cdk?

        Show
        cos Konstantin Boudnik added a comment - what's cdk?
        Hide
        bmahe Bruno Mahé added a comment -

        Cloudera Development Kit - http://cloudera.github.io/cdk/docs/current/
        It includes a very nice framework for writing processing streams.

        See also http://cloudera.github.io/cdk/docs/current/cdk-morphlines/index.html and http://flume.apache.org/FlumeUserGuide.html#morphline-interceptor for more information about morphlines.

        Show
        bmahe Bruno Mahé added a comment - Cloudera Development Kit - http://cloudera.github.io/cdk/docs/current/ It includes a very nice framework for writing processing streams. See also http://cloudera.github.io/cdk/docs/current/cdk-morphlines/index.html and http://flume.apache.org/FlumeUserGuide.html#morphline-interceptor for more information about morphlines.
        Hide
        cos Konstantin Boudnik added a comment -

        Is it in Apache?

        Show
        cos Konstantin Boudnik added a comment - Is it in Apache?
        Hide
        mackrorysd Sean Mackrory added a comment -

        ASL? Yes. ASF? No. Same boat as Hue, in that regard.

        Show
        mackrorysd Sean Mackrory added a comment - ASL? Yes. ASF? No. Same boat as Hue, in that regard.
        Hide
        cos Konstantin Boudnik added a comment -

        Well... there was that already. There was a huge fight over Hue before and I don't want to have it again. I would a strong -1 on non-ASF components' inclusion, as it leads to all sorts of unintended consequences. Yes, it might be very useful, but unless it is an open-community we can't afford it.

        Show
        cos Konstantin Boudnik added a comment - Well... there was that already. There was a huge fight over Hue before and I don't want to have it again. I would a strong -1 on non-ASF components' inclusion, as it leads to all sorts of unintended consequences. Yes, it might be very useful, but unless it is an open-community we can't afford it.
        Hide
        bmahe Bruno Mahé added a comment - - edited

        I am not sure to follow at all the reasoning:

        • Isn't a vote supposed to be technically motivated? I am not sure stating that you don't want to deal with building consensus yourself warrant that.
        • Also, you don't even have to deal with building the consensus. You can always sit on the side and watch (or not). Also the issue with regards to hue was created by only one person, and the issues seemed to me rather political.
        • There was no such issue for datafu or phoenix. There is no reason to have any issue with CDK as well. So let's not give some individuals more credit than they deserve.
        • If we go down the route to exclude any non-ASF software, then let's remove phoenix, hue and datafu. We cannot have the cake and eat it too. We cannot selectively apply such rule based on affinities.
        • What are the unintended consequences for Hue? What were they for phoenix and datafu? It is way too vague to help building consensus. I cannot recall any unintended consequence related to these projects.
        • And finally, right now, there is no issue with it. So I don't see why we should censor ourselves in fear of someone who might stir some troubles. No reason to look for issues when there is none.
        Show
        bmahe Bruno Mahé added a comment - - edited I am not sure to follow at all the reasoning: Isn't a vote supposed to be technically motivated? I am not sure stating that you don't want to deal with building consensus yourself warrant that. Also, you don't even have to deal with building the consensus. You can always sit on the side and watch (or not). Also the issue with regards to hue was created by only one person, and the issues seemed to me rather political. There was no such issue for datafu or phoenix. There is no reason to have any issue with CDK as well. So let's not give some individuals more credit than they deserve. If we go down the route to exclude any non-ASF software, then let's remove phoenix, hue and datafu. We cannot have the cake and eat it too. We cannot selectively apply such rule based on affinities. What are the unintended consequences for Hue? What were they for phoenix and datafu? It is way too vague to help building consensus. I cannot recall any unintended consequence related to these projects. And finally, right now, there is no issue with it. So I don't see why we should censor ourselves in fear of someone who might stir some troubles. No reason to look for issues when there is none.
        Hide
        cos Konstantin Boudnik added a comment -

        Bruno, I think you've read to much into my short comment I will essentially skip over first 5 bullet-points in your comment, because they are exactly emotional and not technical.

        The unintended consequences for Hue and Flume are that this community had to deal with all the issues in the product in order to avoid breaking the stack. E.g. BIGTOP-1134, BIGTOP-1146 at least from most recent times.

        I won't address Phoenix matter as I don't know enough about it. Perhaps this is a great thing to have. However, it hasn't been around for too long, so I can make any informed decision about it.

        The technical issue is what I have in mind first and foremost, because CDK isn't an open-community project and is developed completely behind the closed doors. I don't see a way we can reach out and influence a solution of any issues they will sooner or later introduce into the product. The reason I am so pessimistic about it is that CDK dev team will always have "business priorities" forced upon them and it never be community driven. This is a critical piece of the company's platform offering that designed to increase the developers base around CDH. In other words - they will be looking after their own priorities, not ours. What if CDH will be phase out? Or made not so open in the future?

        Show
        cos Konstantin Boudnik added a comment - Bruno, I think you've read to much into my short comment I will essentially skip over first 5 bullet-points in your comment, because they are exactly emotional and not technical. The unintended consequences for Hue and Flume are that this community had to deal with all the issues in the product in order to avoid breaking the stack. E.g. BIGTOP-1134 , BIGTOP-1146 at least from most recent times. I won't address Phoenix matter as I don't know enough about it. Perhaps this is a great thing to have. However, it hasn't been around for too long, so I can make any informed decision about it. The technical issue is what I have in mind first and foremost, because CDK isn't an open-community project and is developed completely behind the closed doors. I don't see a way we can reach out and influence a solution of any issues they will sooner or later introduce into the product. The reason I am so pessimistic about it is that CDK dev team will always have "business priorities" forced upon them and it never be community driven. This is a critical piece of the company's platform offering that designed to increase the developers base around CDH. In other words - they will be looking after their own priorities, not ours. What if CDH will be phase out? Or made not so open in the future?
        Hide
        rvs Roman Shaposhnik added a comment -

        Guys, let me chime in with my perspective: first of all, I've been involved with the morphlines side of CDK (which is the lion share of functionality I'm interested in there to begin with) and worse comes to worse I'm pretty sure I can support it well enough. Essentially, because of my personal interest in technology I'd be willing to sign up not only for packaging support but also integration with the rest of Bigtop's components. I feel pretty pretty confident that I can get patches in.

        On top of that, I believe we're already shipping the jars files of CDK in solr packages. This is definitely a bug from a packaging perspective (something I've been meaning to clean up for Bigtop 0.8.0). solr packages have no business re-shipping hadoop, CDK, etc. jars – they need to bind to them the same way we do it everywhere else. From that perspective I'd consider this very JIRA a bugfix, rather than adding new functionality (that ship has sailed when solr got included).

        Finally, this is software right? Even if we include CDK today doesn't mean we can't revert the decision if and when we get burnt. For example, if I get strong, non-technical pushback on my fixes in the upstream CDK repo – I'd be the first one to start a discussion of either forking it or dropping alltogether

        Show
        rvs Roman Shaposhnik added a comment - Guys, let me chime in with my perspective: first of all, I've been involved with the morphlines side of CDK (which is the lion share of functionality I'm interested in there to begin with) and worse comes to worse I'm pretty sure I can support it well enough. Essentially, because of my personal interest in technology I'd be willing to sign up not only for packaging support but also integration with the rest of Bigtop's components. I feel pretty pretty confident that I can get patches in. On top of that, I believe we're already shipping the jars files of CDK in solr packages. This is definitely a bug from a packaging perspective (something I've been meaning to clean up for Bigtop 0.8.0). solr packages have no business re-shipping hadoop, CDK, etc. jars – they need to bind to them the same way we do it everywhere else. From that perspective I'd consider this very JIRA a bugfix, rather than adding new functionality (that ship has sailed when solr got included). Finally, this is software right? Even if we include CDK today doesn't mean we can't revert the decision if and when we get burnt. For example, if I get strong, non-technical pushback on my fixes in the upstream CDK repo – I'd be the first one to start a discussion of either forking it or dropping alltogether
        Hide
        bmahe Bruno Mahé added a comment -

        You didn't give much to work with in the first place
        So I worked with only 2 arguments you gave for the veto:

        • There was a huge fight over Hue before and I don't want to have it again

        • it leads to all sorts of unintended consequences

        Unfortunately I still disagree with your arguments.
        First of all the tickets listed proved these issues could be fixed very easily. So it does not seem to be such a big issue that we cannot afford to include CDK. But I also fail to understand why you picked these in example since Apache Flume is an Apache project and an open community.
        Then all the issues you cite are applicable as well to ASF projects (see the 2 issues you cite yourself). Some communities may not have goals aligned with us and make it difficult for us to work with them. For instance communities who did not want to support Apache Hadoop 2 when we needed it or some of the patch we had some troubles to send them.
        Note that CDK is not developed completely behind closed doors since all commits are pushed into its publicly available repository: https://github.com/cloudera/cdk
        And I would not worry about what will happen in the future. If CDK stopped being maintained, then we can always remove it then. And again a similar situation could happen whenever an Apache project has a dying community or goes to the attic. So nothing special about CDK.

        And again, if your arguments hold against CDK, they ought to hold against Hue, Phoenix and datafu.
        Nothing bad has happened for these projects that we had to remove them and I don't see why this would not be true for CDK as well.

        Maybe it would make it easier if you could explain what is so different in CDK from Hue, Phoenix and datafu that makes it unaffordable for us to even include it?

        Show
        bmahe Bruno Mahé added a comment - You didn't give much to work with in the first place So I worked with only 2 arguments you gave for the veto: There was a huge fight over Hue before and I don't want to have it again it leads to all sorts of unintended consequences Unfortunately I still disagree with your arguments. First of all the tickets listed proved these issues could be fixed very easily. So it does not seem to be such a big issue that we cannot afford to include CDK. But I also fail to understand why you picked these in example since Apache Flume is an Apache project and an open community. Then all the issues you cite are applicable as well to ASF projects (see the 2 issues you cite yourself). Some communities may not have goals aligned with us and make it difficult for us to work with them. For instance communities who did not want to support Apache Hadoop 2 when we needed it or some of the patch we had some troubles to send them. Note that CDK is not developed completely behind closed doors since all commits are pushed into its publicly available repository: https://github.com/cloudera/cdk And I would not worry about what will happen in the future. If CDK stopped being maintained, then we can always remove it then. And again a similar situation could happen whenever an Apache project has a dying community or goes to the attic. So nothing special about CDK. And again, if your arguments hold against CDK, they ought to hold against Hue, Phoenix and datafu. Nothing bad has happened for these projects that we had to remove them and I don't see why this would not be true for CDK as well. Maybe it would make it easier if you could explain what is so different in CDK from Hue, Phoenix and datafu that makes it unaffordable for us to even include it?
        Hide
        cos Konstantin Boudnik added a comment -

        I feel pretty pretty confident that I can get patches in.

        I guess that is as much as I need to hear about this component then

        Show
        cos Konstantin Boudnik added a comment - I feel pretty pretty confident that I can get patches in. I guess that is as much as I need to hear about this component then
        Hide
        mackrorysd Sean Mackrory added a comment -

        Attaching a fairly straight-forward port from the upstream packaging as a possible starting point for any further work or discussion. I've created DEBs on Ubuntu 10.04 and RPMs on OpenSuSE and the contents look correct (in terms of matching the official packages).

        I do notice the packages include all the -javadoc, -sources, and -tests JARs for CDK and I don't think they should (at least not in the form they do). There were also some specific JDK 6 and Maven 3 version requirements that may not be entirely necessary and might need to be removed / overridden for all the distros we're targeting.

        Show
        mackrorysd Sean Mackrory added a comment - Attaching a fairly straight-forward port from the upstream packaging as a possible starting point for any further work or discussion. I've created DEBs on Ubuntu 10.04 and RPMs on OpenSuSE and the contents look correct (in terms of matching the official packages). I do notice the packages include all the -javadoc, -sources, and -tests JARs for CDK and I don't think they should (at least not in the form they do). There were also some specific JDK 6 and Maven 3 version requirements that may not be entirely necessary and might need to be removed / overridden for all the distros we're targeting.
        Hide
        bmahe Bruno Mahé added a comment -

        Thanks a lot!
        It looks great but some of the packages in the patch require avro-libs, which is not provided in Apache Bigtop or standards repositories.

        Show
        bmahe Bruno Mahé added a comment - Thanks a lot! It looks great but some of the packages in the patch require avro-libs, which is not provided in Apache Bigtop or standards repositories.
        Hide
        mackrorysd Sean Mackrory added a comment -

        Ah yes - the portion of the original packaging that depends on that was already removed, so I'll drop the dependency as well (at the very least until Bigtop packages Avro, but it may still be irrelevant then).

        Also, CDK was renamed to Kite yesterday and is being re-released as Kite 0.10.0. As soon as there is a tarball of said release, I'll update my patch accordingly.

        Show
        mackrorysd Sean Mackrory added a comment - Ah yes - the portion of the original packaging that depends on that was already removed, so I'll drop the dependency as well (at the very least until Bigtop packages Avro, but it may still be irrelevant then). Also, CDK was renamed to Kite yesterday and is being re-released as Kite 0.10.0. As soon as there is a tarball of said release, I'll update my patch accordingly.
        Hide
        apurtell Andrew Purtell added a comment -

        Phoenix is very likely to be incubating at Apache in a week or two.

        +1 to a contribution of Kite, provided:

        • As with the other non-ASF code we have taken, there is a public open source repository available that is accepting contributions or issues, a GitHub repo with issues enabled is a good example
        • There are no dependencies on non-public non-open source.
        • Its licensing is congruent with the rest of the stack/components we have, the ASL / BSD side of the family.
        • There are no web bugs in the packaging or documentation
        Show
        apurtell Andrew Purtell added a comment - Phoenix is very likely to be incubating at Apache in a week or two. +1 to a contribution of Kite, provided: As with the other non-ASF code we have taken, there is a public open source repository available that is accepting contributions or issues, a GitHub repo with issues enabled is a good example There are no dependencies on non-public non-open source. Its licensing is congruent with the rest of the stack/components we have, the ASL / BSD side of the family. There are no web bugs in the packaging or documentation
        Hide
        esammer E. Sammer added a comment -

        Hey all. I manage the Kite team at Cloudera. I just wanted to chime in and say that I'd love to see Kite included in Bigtop. We renamed it from the CDK to indicate that it's not Cloudera-specific, and really a community project. We are very interested in non-Cloudera contribution, in all forms, to Kite (and have taken them in the past). The project is open to, and interested in, giving community members commit bit on Kite. We're also fully committed to the ASF license for Kite. Kite will never depend on proprietary code.

        I understand the consternation with including support for non-ASF projects in Bigtop, but I want to work with everyone to do what we can to make this possible. I think the benefits of inclusion outweigh any perceived risk, and hope that previous "spirited debates" don't preclude the consideration of new projects.

        Show
        esammer E. Sammer added a comment - Hey all. I manage the Kite team at Cloudera. I just wanted to chime in and say that I'd love to see Kite included in Bigtop. We renamed it from the CDK to indicate that it's not Cloudera-specific, and really a community project. We are very interested in non-Cloudera contribution, in all forms, to Kite (and have taken them in the past). The project is open to, and interested in, giving community members commit bit on Kite. We're also fully committed to the ASF license for Kite. Kite will never depend on proprietary code. I understand the consternation with including support for non-ASF projects in Bigtop, but I want to work with everyone to do what we can to make this possible. I think the benefits of inclusion outweigh any perceived risk, and hope that previous "spirited debates" don't preclude the consideration of new projects.
        Hide
        mackrorysd Sean Mackrory added a comment -

        Here's a patch with the rename and the erroneous dependency taken into account. Again - I haven't tested functionality as I haven't worked with Kite, just that the contents looks correct. We'll want to add some package tests to this, and maybe some code for components like Flume to include this in the classpath if it's installed.

        Show
        mackrorysd Sean Mackrory added a comment - Here's a patch with the rename and the erroneous dependency taken into account. Again - I haven't tested functionality as I haven't worked with Kite, just that the contents looks correct. We'll want to add some package tests to this, and maybe some code for components like Flume to include this in the classpath if it's installed.
        Hide
        bmahe Bruno Mahé added a comment -

        Eric> Thanks a lot! I really appreciate it.

        Sean> Will test this over the week end.
        Thanks a lot for the patch and everything! I really appreciate it as well.

        Regarding Apache Flume integration, I installed CDK/Kite in the /usr/lib/flume-ng/plugins.d/cdk/libext and Apache Flume was happy. There was no need to edit any file or any config since the directory is picked up automatically.
        So what about a flume-kite package which depends on flume-ng and sets up everything under /usr/lib/flume-ng/plugins.d/kite/libext? We can look into this later on after this ticket is resolved.

        Show
        bmahe Bruno Mahé added a comment - Eric> Thanks a lot! I really appreciate it. Sean> Will test this over the week end. Thanks a lot for the patch and everything! I really appreciate it as well. Regarding Apache Flume integration, I installed CDK/Kite in the /usr/lib/flume-ng/plugins.d/cdk/libext and Apache Flume was happy. There was no need to edit any file or any config since the directory is picked up automatically. So what about a flume-kite package which depends on flume-ng and sets up everything under /usr/lib/flume-ng/plugins.d/kite/libext? We can look into this later on after this ticket is resolved.
        Hide
        bmahe Bruno Mahé added a comment -

        I have not forgotten about this ticket. Will get to it as soon as I can.
        Sorry for the delay.

        Show
        bmahe Bruno Mahé added a comment - I have not forgotten about this ticket. Will get to it as soon as I can. Sorry for the delay.
        Hide
        bmahe Bruno Mahé added a comment -

        Finally got to try but it failed to build.
        Unfortenately it fails because Kite use a maven enforcer to force the use of JDK6:

        Detected JDK Version: 1.7.0-45 is not in the allowed range [1.6.0,1.6.1000}].
        

        But when I point my java home to JDK 6, then maven fails to start since it was build for a newer jvm.
        So I will have to download another version of maven.

        Something I noticed also, but not sure if it is intended or not:

         cp -r LICENSE.txt NOTICE.txt README.md build kite-data kite-flume-avro-event-serializer kite-maven-plugin kite-morphlines kite-tools lib pom.xml src build/kite-0.10.0
        cp: cannot copy a directory, 'build', into itself, 'build/kite-0.10.0/build'
        
        Show
        bmahe Bruno Mahé added a comment - Finally got to try but it failed to build. Unfortenately it fails because Kite use a maven enforcer to force the use of JDK6: Detected JDK Version: 1.7.0-45 is not in the allowed range [1.6.0,1.6.1000}]. But when I point my java home to JDK 6, then maven fails to start since it was build for a newer jvm. So I will have to download another version of maven. Something I noticed also, but not sure if it is intended or not: cp -r LICENSE.txt NOTICE.txt README.md build kite-data kite-flume-avro-event-serializer kite-maven-plugin kite-morphlines kite-tools lib pom.xml src build/kite-0.10.0 cp: cannot copy a directory, 'build', into itself, 'build/kite-0.10.0/build'
        Hide
        bmahe Bruno Mahé added a comment -

        Hold on. Setting -DjavaVersion=1.7 helps me going forward without getting another Apache Maven

        Show
        bmahe Bruno Mahé added a comment - Hold on. Setting -DjavaVersion=1.7 helps me going forward without getting another Apache Maven
        Hide
        bmahe Bruno Mahé added a comment -

        It builds RPM fine for me.
        I noticed jars from Apache Solr and Zookeeper are included and not symlinked. But this can be fixed in another ticket.

        I will try to see if deb packages build fine tomorrow.

        Show
        bmahe Bruno Mahé added a comment - It builds RPM fine for me. I noticed jars from Apache Solr and Zookeeper are included and not symlinked. But this can be fixed in another ticket. I will try to see if deb packages build fine tomorrow.
        Hide
        rvs Roman Shaposhnik added a comment -

        A few nits:

        • Copyright: 2013, Cloudera, Inc – needs to be changed to ASF Copyright
        • please add at least rudimentary package tests

        Finally, since Kite is not really a deployed component – I don't think we would require smoke tests, but if those are easy to provide – we should probably look into that.

        Show
        rvs Roman Shaposhnik added a comment - A few nits: Copyright: 2013, Cloudera, Inc – needs to be changed to ASF Copyright please add at least rudimentary package tests Finally, since Kite is not really a deployed component – I don't think we would require smoke tests, but if those are easy to provide – we should probably look into that.
        Hide
        cos Konstantin Boudnik added a comment -

        Any activity is planned here? If not - let's move on.

        Show
        cos Konstantin Boudnik added a comment - Any activity is planned here? If not - let's move on.
        Hide
        warwithin YoungWoo Kim added a comment -

        Bruno Mahé Any progress on this? I've been using Kite especially Morphlines. It's very useful for my projects.

        If you don't mind I can rebase and add tests based on your patch. Please let me know.

        Show
        warwithin YoungWoo Kim added a comment - Bruno Mahé Any progress on this? I've been using Kite especially Morphlines. It's very useful for my projects. If you don't mind I can rebase and add tests based on your patch. Please let me know.
        Hide
        rvs Roman Shaposhnik added a comment -

        YoungWoo Kim I think the real question is: who'd be maintaining Kite in Bigtop long term. It would be great if you could volunteer.

        Show
        rvs Roman Shaposhnik added a comment - YoungWoo Kim I think the real question is: who'd be maintaining Kite in Bigtop long term. It would be great if you could volunteer.
        Hide
        warwithin YoungWoo Kim added a comment -

        Roman Shaposhnik NP, I would like to do that.

        Show
        warwithin YoungWoo Kim added a comment - Roman Shaposhnik NP, I would like to do that.
        Hide
        rdblue Ryan Blue added a comment -

        YoungWoo Kim thanks! Let us know how the Kite team can help this effort and if we need to make any changes upstream.

        Show
        rdblue Ryan Blue added a comment - YoungWoo Kim thanks! Let us know how the Kite team can help this effort and if we need to make any changes upstream.
        Hide
        bmahe Bruno Mahé added a comment -

        YoungWoo Kim Thanks for volunteering!
        Unfortunately I did not have time to test on a debian VM. So please, go ahead, rebase and improve the patches.
        But these are not my patches, they are Sean's.

        Show
        bmahe Bruno Mahé added a comment - YoungWoo Kim Thanks for volunteering! Unfortunately I did not have time to test on a debian VM. So please, go ahead, rebase and improve the patches. But these are not my patches, they are Sean's.
        Hide
        warwithin YoungWoo Kim added a comment -

        Bruno Mahé Got it. thanks for your comments.

        Show
        warwithin YoungWoo Kim added a comment - Bruno Mahé Got it. thanks for your comments.
        Hide
        warwithin YoungWoo Kim added a comment -

        Ryan Blue Thanks for your comments. I'm working on packaging bits for Kite 0.17.1

        Actually I got a issue on building Kite [1]. Kite minicluster pulls dependencies from CDH but should not in Bigtop. first time I removed minicluster bits from packaging however Kite minicluster is just awesome so I would like to include it in Bigtop.

        It would be very nice if the issue is resolved in upstream.

        1. https://groups.google.com/a/cloudera.org/forum/#!topic/cdk-dev/oAtIbL_LB0c

        Show
        warwithin YoungWoo Kim added a comment - Ryan Blue Thanks for your comments. I'm working on packaging bits for Kite 0.17.1 Actually I got a issue on building Kite [1] . Kite minicluster pulls dependencies from CDH but should not in Bigtop. first time I removed minicluster bits from packaging however Kite minicluster is just awesome so I would like to include it in Bigtop. It would be very nice if the issue is resolved in upstream. 1. https://groups.google.com/a/cloudera.org/forum/#!topic/cdk-dev/oAtIbL_LB0c
        Hide
        warwithin YoungWoo Kim added a comment - - edited

        Unfortunately Kite 0.17.1 does not build with Hive 0.14.0. See HIVE-8906. AFAIK Hive team will release next version of Hive – 0.14.1 1.0 and 0.15.0 1.1. Once the next version is released, I'll update a patch for Kite.

        Show
        warwithin YoungWoo Kim added a comment - - edited Unfortunately Kite 0.17.1 does not build with Hive 0.14.0. See HIVE-8906 . AFAIK Hive team will release next version of Hive – 0.14.1 1.0 and 0.15.0 1.1. Once the next version is released, I'll update a patch for Kite.
        Hide
        warwithin YoungWoo Kim added a comment - - edited

        Bruno Mahé I update the summary for this jira.

        Show
        warwithin YoungWoo Kim added a comment - - edited Bruno Mahé I update the summary for this jira.
        Hide
        bmahe Bruno Mahé added a comment -

        Thanks!

        Show
        bmahe Bruno Mahé added a comment - Thanks!
        Hide
        warwithin YoungWoo Kim added a comment - - edited

        Updated patch (BIGTOP-1149.1.patch):

        • exclude kite-minicluster
        • add a smoke test for kite-dataset

        Before you run the test, make sure hadoop hdfs services are running.

        export KITE_HOME=/usr/lib/kite
        cd bigtop-tests/smoke-tests/
        ./gradlew compileGroovy test -Dsmoke.tests=kite --info
        
        Show
        warwithin YoungWoo Kim added a comment - - edited Updated patch ( BIGTOP-1149 .1.patch): exclude kite-minicluster add a smoke test for kite-dataset Before you run the test, make sure hadoop hdfs services are running. export KITE_HOME=/usr/lib/kite cd bigtop-tests/smoke-tests/ ./gradlew compileGroovy test -Dsmoke.tests=kite --info
        Hide
        warwithin YoungWoo Kim added a comment -

        Updated patch, BIGTOP-1149.2.patch:

        • Fix deb packaging
        • Add versionless symlinks for kite-* jar files
        Show
        warwithin YoungWoo Kim added a comment - Updated patch, BIGTOP-1149 .2.patch: Fix deb packaging Add versionless symlinks for kite-* jar files
        Hide
        warwithin YoungWoo Kim added a comment -

        Updated patch, BIGTOP-1149.3.patch:

        • minor fix
        Show
        warwithin YoungWoo Kim added a comment - Updated patch, BIGTOP-1149 .3.patch: minor fix
        Hide
        warwithin YoungWoo Kim added a comment -

        Updated patch (BIGTOP-1149.4.patch):

        • Remove duplicated jars
        • Add a symlink for zookeeper jar
        • Update MAINTAINERS.txt for Kite
        Show
        warwithin YoungWoo Kim added a comment - Updated patch ( BIGTOP-1149 .4.patch): Remove duplicated jars Add a symlink for zookeeper jar Update MAINTAINERS.txt for Kite
        Hide
        warwithin YoungWoo Kim added a comment -
        Show
        warwithin YoungWoo Kim added a comment - BIGTOP-1149.5.patch Kite 1.0.0
        Hide
        warwithin YoungWoo Kim added a comment -

        BIGTOP-1149.6.patch

        • Revised patch for Kite 1.0.0
        Show
        warwithin YoungWoo Kim added a comment - BIGTOP-1149.6.patch Revised patch for Kite 1.0.0
        Hide
        warwithin YoungWoo Kim added a comment -

        BIGTOP-1149.7.patch

        • Rebase against master
        Show
        warwithin YoungWoo Kim added a comment - BIGTOP-1149.7.patch Rebase against master
        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user youngwookim opened a pull request:

        https://github.com/apache/bigtop/pull/22

        BIGTOP-1149: Package Kite

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/youngwookim/bigtop BIGTOP-1149

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/bigtop/pull/22.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #22


        commit eb5de1db8c51899a321f863207f59e006945d5af
        Author: Youngwoo Kim <warwithin@gmail.com>
        Date: 2015-01-22T07:47:29Z

        BIGTOP-1149: Package Kite


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user youngwookim opened a pull request: https://github.com/apache/bigtop/pull/22 BIGTOP-1149 : Package Kite You can merge this pull request into a Git repository by running: $ git pull https://github.com/youngwookim/bigtop BIGTOP-1149 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/22.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22 commit eb5de1db8c51899a321f863207f59e006945d5af Author: Youngwoo Kim <warwithin@gmail.com> Date: 2015-01-22T07:47:29Z BIGTOP-1149 : Package Kite
        Hide
        warwithin YoungWoo Kim added a comment - - edited

        I just revised the patch and send a PR:

        • Bump up Kite version to 1.1.0
        • Revise scripts

        I believe it's ready to review. Thanks!

        Show
        warwithin YoungWoo Kim added a comment - - edited I just revised the patch and send a PR: Bump up Kite version to 1.1.0 Revise scripts I believe it's ready to review. Thanks!
        Hide
        evans_ye Evans Ye added a comment -

        Sorry I've zero knowledge in Kite so probably other experienced Committer such as Bruno Mahé can help, maybe?

        Show
        evans_ye Evans Ye added a comment - Sorry I've zero knowledge in Kite so probably other experienced Committer such as Bruno Mahé can help, maybe?
        Hide
        bmahe Bruno Mahé added a comment -

        Thanks for the patch!
        I will take a look

        Show
        bmahe Bruno Mahé added a comment - Thanks for the patch! I will take a look
        Hide
        bmahe Bruno Mahé added a comment -

        I just looked at the patch and everything looks good.
        Next step is to do a build.

        Show
        bmahe Bruno Mahé added a comment - I just looked at the patch and everything looks good. Next step is to do a build.
        Hide
        bmahe Bruno Mahé added a comment -

        I tried to build Kite but got the following error:

        [WARNING] Rule 1: org.apache.maven.plugins.enforcer.RequireJavaVersion failed with message:
        Detected JDK Version: 1.8.0-51 is not in the allowed range [1.7.0,1.7.1000}].
        ...
        ...
        [INFO] ------------------------------------------------------------------------
        [INFO] BUILD FAILURE
        [INFO] ------------------------------------------------------------------------
        [INFO] Total time: 44.270 s
        [INFO] Finished at: 2015-07-26T23:35:51-07:00
        [INFO] Final Memory: 33M/236M
        [INFO] ------------------------------------------------------------------------
        [ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:1.3.1:enforce (clean) on project kite-parent: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed. -> [Help 1]
        [ERROR] 
        [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
        [ERROR] Re-run Maven using the -X switch to enable full debug logging.
        [ERROR] 
        [ERROR] For more information about the errors and possible solutions, please read the following articles:
        [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
        error: Bad exit status from /var/tmp/rpm-tmp.ykouBi (%build)
            Bad exit status from /var/tmp/rpm-tmp.ykouBi (%build)
        
        
        RPM build errors:
        :kite-rpm FAILED
        
        FAILURE: Build failed with an exception.
        
        * Where:
        Script '/home/bruno/projects/bigtop-asf/packages.gradle' line: 420
        
        * What went wrong:
        Execution failed for task ':kite-rpm'.
        > Process 'command 'rpmbuild'' finished with non-zero exit value 1
        
        * Try:
        Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
        
        BUILD FAILED
        
        Total time: 2 mins 22.333 secs
        

        Looking at https://github.com/kite-sdk/kite/blob/master/pom.xml#L249 I see the following:

          <properties>
            <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        
            <!-- Java versions -->
            <javaVersion>1.7</javaVersion>
        

        and

              <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-enforcer-plugin</artifactId>
                <version>${vers.maven-enforcer-plugin}</version>
                <inherited>false</inherited>
                <configuration>
                  <rules>
                    <requireMavenVersion>
                      <version>[3.0.0,)</version>
                    </requireMavenVersion>
                    <requireJavaVersion>
                      <version>[${javaVersion}.0,${javaVersion}.1000}]</version>
                    </requireJavaVersion>
                  </rules>
                </configuration>
        

        and also

            <profile>
              <id>jdk1.8+</id>
              <activation>
                <jdk>[1.8,)</jdk>
              </activation>
              <properties>                  
                <!-- see http://stackoverflow.com/questions/15886209/maven-is-not-working-in-java-8-when-javadoc-tags-are-incomplete -->
                <Xdoclint>-Xdoclint:none</Xdoclint>
              </properties>
            </profile>
        

        So I am a bit confused regarding that enforce rule.
        Are you aware of any issue with java 8? (but if there was some issue, there would not see any profile?)

        Show
        bmahe Bruno Mahé added a comment - I tried to build Kite but got the following error: [WARNING] Rule 1: org.apache.maven.plugins.enforcer.RequireJavaVersion failed with message: Detected JDK Version: 1.8.0-51 is not in the allowed range [1.7.0,1.7.1000}]. ... ... [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 44.270 s [INFO] Finished at: 2015-07-26T23:35:51-07:00 [INFO] Final Memory: 33M/236M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:1.3.1:enforce (clean) on project kite-parent: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed. -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException error: Bad exit status from /var/tmp/rpm-tmp.ykouBi (%build) Bad exit status from /var/tmp/rpm-tmp.ykouBi (%build) RPM build errors: :kite-rpm FAILED FAILURE: Build failed with an exception. * Where: Script '/home/bruno/projects/bigtop-asf/packages.gradle' line: 420 * What went wrong: Execution failed for task ':kite-rpm'. > Process 'command 'rpmbuild'' finished with non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. BUILD FAILED Total time: 2 mins 22.333 secs Looking at https://github.com/kite-sdk/kite/blob/master/pom.xml#L249 I see the following: <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <!-- Java versions --> <javaVersion>1.7</javaVersion> and <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-enforcer-plugin</artifactId> <version>${vers.maven-enforcer-plugin}</version> <inherited>false</inherited> <configuration> <rules> <requireMavenVersion> <version>[3.0.0,)</version> </requireMavenVersion> <requireJavaVersion> <version>[${javaVersion}.0,${javaVersion}.1000}]</version> </requireJavaVersion> </rules> </configuration> and also <profile> <id>jdk1.8+</id> <activation> <jdk>[1.8,)</jdk> </activation> <properties> <!-- see http://stackoverflow.com/questions/15886209/maven-is-not-working-in-java-8-when-javadoc-tags-are-incomplete --> <Xdoclint>-Xdoclint:none</Xdoclint> </properties> </profile> So I am a bit confused regarding that enforce rule. Are you aware of any issue with java 8? (but if there was some issue, there would not see any profile?)
        Hide
        warwithin YoungWoo Kim added a comment - - edited

        Thanks for looking into this, Bruno Mahé! AFAIK, The default Java(JDK) for current Bigtop stack is 1.7. bigtop.mk:

        # JDK Version
        JDK_VERSION=1.7
        JDK_BASE_VERSION=$(JDK_VERSION)
        

        So I've tested with Java 1.7. I think that upgrading JDK to 1.8 affects rest of the components on Bigtop at this point.

        However, for any reason, If you want to build Kite with JDK 1.8 you should modify the 'do-component-build':

        mvn clean install \
          -DjavaVersion=1.8 \
          -Dvers.hadoop2=${HADOOP_VERSION} \
        
        ...
        
        Show
        warwithin YoungWoo Kim added a comment - - edited Thanks for looking into this, Bruno Mahé ! AFAIK, The default Java(JDK) for current Bigtop stack is 1.7. bigtop.mk: # JDK Version JDK_VERSION=1.7 JDK_BASE_VERSION=$(JDK_VERSION) So I've tested with Java 1.7. I think that upgrading JDK to 1.8 affects rest of the components on Bigtop at this point. However, for any reason, If you want to build Kite with JDK 1.8 you should modify the 'do-component-build': mvn clean install \ -DjavaVersion=1.8 \ -Dvers.hadoop2=${HADOOP_VERSION} \ ...
        Hide
        rdblue Ryan Blue added a comment -

        YoungWoo is right. We added this as a sanity check. It doesn't mean you can't build Kite with 1.8, you just have to explicitly set it. That allows us to control other build settings from the javaVersion too. We actually test Kite with a variety of profiles and 1.8 is one of them.

        Show
        rdblue Ryan Blue added a comment - YoungWoo is right. We added this as a sanity check. It doesn't mean you can't build Kite with 1.8, you just have to explicitly set it. That allows us to control other build settings from the javaVersion too. We actually test Kite with a variety of profiles and 1.8 is one of them.
        Hide
        warwithin YoungWoo Kim added a comment - - edited

        Thanks for your clarification, Ryan Blue!

        Bruno Mahé, building Kite with Java 1.8 works fine on my end but I would encourage you to validate the patch with Java 1.7

        Show
        warwithin YoungWoo Kim added a comment - - edited Thanks for your clarification, Ryan Blue ! Bruno Mahé , building Kite with Java 1.8 works fine on my end but I would encourage you to validate the patch with Java 1.7
        Hide
        warwithin YoungWoo Kim added a comment -

        Hey Bruno Mahé I've just rebased to master. The patch for current master here, https://patch-diff.githubusercontent.com/raw/apache/bigtop/pull/22.patch

        Show
        warwithin YoungWoo Kim added a comment - Hey Bruno Mahé I've just rebased to master. The patch for current master here, https://patch-diff.githubusercontent.com/raw/apache/bigtop/pull/22.patch
        Hide
        bmahe Bruno Mahé added a comment -

        +1
        Great work!

        Can someone integrate the patch or point me to the new github flow? Or is just applying the patch the current recommended way?

        Show
        bmahe Bruno Mahé added a comment - +1 Great work! Can someone integrate the patch or point me to the new github flow? Or is just applying the patch the current recommended way?
        Hide
        warwithin YoungWoo Kim added a comment -

        Thank you for reviewing patch Bruno Mahé! I'll merge the patch shortly.

        Show
        warwithin YoungWoo Kim added a comment - Thank you for reviewing patch Bruno Mahé ! I'll merge the patch shortly.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user asfgit closed the pull request at:

        https://github.com/apache/bigtop/pull/22

        Show
        githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/bigtop/pull/22
        Hide
        warwithin YoungWoo Kim added a comment -

        Committed!

        Show
        warwithin YoungWoo Kim added a comment - Committed!

          People

          • Assignee:
            warwithin YoungWoo Kim
            Reporter:
            bmahe Bruno Mahé
          • Votes:
            0 Vote for this issue
            Watchers:
            14 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development