HBase
  1. HBase
  2. HBASE-3374

Our jruby jar has *GPL jars in it; fix

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.92.0
    • Component/s: None
    • Labels:
      None
    • Release Note:
      The lads over in jruby land say 1.6RC2 should have the licensing fixup. Lets get it into TRUNK when ready. I don't think this a 0.90.x issue. Moving it out.

      Description

      The latest JRuby's complete jar bundles *GPL jars (JNA and JFFI among others). It looks like the functionality we depend on – the shell in particular – makes use of these dirty jars so they are hard to strip. They came in because we (I!) just updated our JRuby w/o checking in on what updates contained. JRuby has been doing this for a while now (1.1.x added the first LGPL). You have to go all the ways back to the original HBase checkin, HBASE-487, of JRuby – 1.0.3 – to get a JRuby w/o *GPL jars.

      Plan is to try and revert our JRuby all the ways down to 1.0.3 before shipping 0.90.0. Thats what this issue is about.

      We should also look into moving off JRuby in the medium to long-term. Its kinda awkward sticking on an old version that is no longer supported. I'll open an issue for that.

        Activity

        stack created issue -
        stack made changes -
        Field Original Value New Value
        Assignee stack [ stack ]
        Hide
        Lars Francke added a comment -

        I'm sorry. I might have been the one who upgraded that in one of the Maven cleanups without taking a close look.

        Show
        Lars Francke added a comment - I'm sorry. I might have been the one who upgraded that in one of the Maven cleanups without taking a close look.
        Hide
        stack added a comment -

        @Lars It was not you. The damage was already done many times over.

        Show
        stack added a comment - @Lars It was not you. The damage was already done many times over.
        Hide
        stack added a comment -

        So, 1.0.3. is still available in maven repos. Thats good. java_import is not available in 1.0.3. Thats the 'safe' way of doing imports — see http://jira.codehaus.org/browse/JRUBY-3171. Using raw 'import' seems to work but dumps out the following before shell starts if the referenced import is used more than once – e.g. if we're using HConstants defines in a few places in .rb code:

        (eval):1 warning: already initialized constant HConstants
        

        Fully specifying java classes with packages in all places they are used seems to get rid of the above. I'm working on a patch to do that.

        We will have to remove TestShell. It uses facility only available in later jrubys.

        Looking at the way the shell was refactored by Alexei, its kinda nice the way he did it. All of the java access is done in one or two files apart from the bulk of the ruby code; makes it easier doing this fixup.

        Show
        stack added a comment - So, 1.0.3. is still available in maven repos. Thats good. java_import is not available in 1.0.3. Thats the 'safe' way of doing imports — see http://jira.codehaus.org/browse/JRUBY-3171 . Using raw 'import' seems to work but dumps out the following before shell starts if the referenced import is used more than once – e.g. if we're using HConstants defines in a few places in .rb code: (eval):1 warning: already initialized constant HConstants Fully specifying java classes with packages in all places they are used seems to get rid of the above. I'm working on a patch to do that. We will have to remove TestShell. It uses facility only available in later jrubys. Looking at the way the shell was refactored by Alexei, its kinda nice the way he did it. All of the java access is done in one or two files apart from the bulk of the ruby code; makes it easier doing this fixup.
        Hide
        stack added a comment -

        Here is what I applied to 0.90. I applied same to trunk but took a little fixup.

        It looks to be all working. I exercised a bunch of shell facility and I think I got all the weird imports and references. Lets open new issues if there is anything that I forgot... Shell seems fine. Maybe a little slower than it used to be.

        Show
        stack added a comment - Here is what I applied to 0.90. I applied same to trunk but took a little fixup. It looks to be all working. I exercised a bunch of shell facility and I think I got all the weird imports and references. Lets open new issues if there is anything that I forgot... Shell seems fine. Maybe a little slower than it used to be.
        stack made changes -
        Attachment jruby.txt [ 12466672 ]
        Hide
        stack added a comment -

        I did a bit more testing of shell.. enable, disable, create, alter, etc. Seems to be doing like it used to.

        Show
        stack added a comment - I did a bit more testing of shell.. enable, disable, create, alter, etc. Seems to be doing like it used to.
        stack made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #1697 (See https://hudson.apache.org/hudson/job/HBase-TRUNK/1697/)

        Show
        Hudson added a comment - Integrated in HBase-TRUNK #1697 (See https://hudson.apache.org/hudson/job/HBase-TRUNK/1697/ )
        Hide
        Stephen Watt added a comment -

        Thanks for turning this issue around so quickly. Awesome work.

        Show
        Stephen Watt added a comment - Thanks for turning this issue around so quickly. Awesome work.
        Hide
        Charles Oliver Nutter added a comment -

        Woah there cowboys!

        Downgrading to JRuby 1.0.3 is a TERRIBLE idea. 1.0.3 does not include a compiler, and performs somewhere in the neighborhood of 50-100 times worse than current JRuby versions. It also has much worse compatibility with Ruby, is not compatible at all with Ruby 1.8.7 or 1.9, and has many other leaks, bugs, and flaws.

        We would happily work with you guys (and Apache proper) to either publish a version of JRuby that does not include the offending libraries, or try to get the creator of those libraries (who is a JRuby committer) to relicense them.

        I would STRONGLY recommend undoing the massive damage you will cause by downgrading to JRuby 1.0.3 and working with us to get you a "safe" version of a more current JRuby.

        Show
        Charles Oliver Nutter added a comment - Woah there cowboys! Downgrading to JRuby 1.0.3 is a TERRIBLE idea. 1.0.3 does not include a compiler, and performs somewhere in the neighborhood of 50-100 times worse than current JRuby versions. It also has much worse compatibility with Ruby, is not compatible at all with Ruby 1.8.7 or 1.9, and has many other leaks, bugs, and flaws. We would happily work with you guys (and Apache proper) to either publish a version of JRuby that does not include the offending libraries, or try to get the creator of those libraries (who is a JRuby committer) to relicense them. I would STRONGLY recommend undoing the massive damage you will cause by downgrading to JRuby 1.0.3 and working with us to get you a "safe" version of a more current JRuby.
        Hide
        ryan rawson added a comment -

        hi Charles,

        The patch has already been submitted, the deed is done.

        In our survey of the code, there were about 4 different libraries embedded and included in JRuby, some of LGPL some of GPLv3.

        Some of them look pretty fundamental, such as the dependency on: http://code.google.com/p/jvm-language-runtime

        As you probably know, the ASF has pretty strict rules regarding licensing, so undoing the downgrade is not possible until a JRuby that has ASF compatible licensing is available.

        Show
        ryan rawson added a comment - hi Charles, The patch has already been submitted, the deed is done. In our survey of the code, there were about 4 different libraries embedded and included in JRuby, some of LGPL some of GPLv3. Some of them look pretty fundamental, such as the dependency on: http://code.google.com/p/jvm-language-runtime As you probably know, the ASF has pretty strict rules regarding licensing, so undoing the downgrade is not possible until a JRuby that has ASF compatible licensing is available.
        Hide
        ryan rawson added a comment -

        Based on COPYING from:
        https://github.com/jruby/jruby/blob/master/COPYING

        The following libraries are a problem:

        build_lib/jffi*jar (http://kenai.com/projects/jffi) and
        build_lib/jgrapht-jdk1.5.jar (http://jgrapht.sourceforge.net) are
        distributed under the GPL v3.

        build_lib/jna*jar (http://jna.dev.java.net) are distributed
        under the LGPL-2.1+ license.

        build_lib/invokedynamic.jar and build_lib/jsr292-mock.jar
        (http://code.google.com/p/jvm-language-runtime) are distributed
        under the LGPL license.

        Also there are a few broken links such as http://projectkenai.com/projects/jaffl which are a little concerning.

        Show
        ryan rawson added a comment - Based on COPYING from: https://github.com/jruby/jruby/blob/master/COPYING The following libraries are a problem: build_lib/jffi*jar ( http://kenai.com/projects/jffi ) and build_lib/jgrapht-jdk1.5.jar ( http://jgrapht.sourceforge.net ) are distributed under the GPL v3. build_lib/jna*jar ( http://jna.dev.java.net ) are distributed under the LGPL-2.1+ license. build_lib/invokedynamic.jar and build_lib/jsr292-mock.jar ( http://code.google.com/p/jvm-language-runtime ) are distributed under the LGPL license. Also there are a few broken links such as http://projectkenai.com/projects/jaffl which are a little concerning.
        Hide
        Charles Oliver Nutter added a comment -

        Ryan: I understand what ASF needs, and that's exactly what I want to provide.

        Of the libraries that are GPL, according to our COPYING file:

        • jvm-language-runtime is not shipped with JRuby binaries. It is used only because it provides a mock JSR-292 invokedynamic library for compile-time on JVMs that do not have JSR-292 (Java 6 and lower).
        • jffi could potentially be relicensed, or simply removed; it is for native library access, and JRuby would work correctly without it.
        • jgrapht-jdk15 is only used by a newer in-development compiler that is not used normally, and could be removed. We would like to explore replacements.
        • we do not ship JNA anymore

        Of these, only jffi and jgrapht are in the binaries we distribute, and it's likely JRuby would work with them just yanked out. If any of the other libraries are included in our binaries, it's a bug in our build process.

        If we provide a build of JRuby that does not include those GPLed libraries, would that satisfy ASF's requirements?

        Show
        Charles Oliver Nutter added a comment - Ryan: I understand what ASF needs, and that's exactly what I want to provide. Of the libraries that are GPL, according to our COPYING file: jvm-language-runtime is not shipped with JRuby binaries. It is used only because it provides a mock JSR-292 invokedynamic library for compile-time on JVMs that do not have JSR-292 (Java 6 and lower). jffi could potentially be relicensed, or simply removed; it is for native library access, and JRuby would work correctly without it. jgrapht-jdk15 is only used by a newer in-development compiler that is not used normally, and could be removed. We would like to explore replacements. we do not ship JNA anymore Of these, only jffi and jgrapht are in the binaries we distribute, and it's likely JRuby would work with them just yanked out. If any of the other libraries are included in our binaries, it's a bug in our build process. If we provide a build of JRuby that does not include those GPLed libraries, would that satisfy ASF's requirements?
        Hide
        Charles Oliver Nutter added a comment -

        FYI, I have taken this to be a JRuby issue, and filed the following: http://jira.codehaus.org/browse/JRUBY-5410

        I want us to publish a *GPL-free maven artifact for JRuby 1.6.0. This should be usable by Apache and other projects that can't include GPLed code.

        I am talking to the maintainer of jffi (and jaffl, which was incorrectly listed as MIT licensed in COPYING) about adding an Apache-friendly license.

        I added a jar-jruby-nogpl target to our build.xml and fixed COPYING to properly reflect jaffl licensing and jffi and jaffl homepages

        I also confirmed that simply removing jffi, jaffl, and jgrapht from jruby.jar does not damage general Ruby behavior. As mentioned above, some native or system-level capabilities are disabled or work differently, but IRB, RubyGems, and Rails all function correctly.

        Show
        Charles Oliver Nutter added a comment - FYI, I have taken this to be a JRuby issue, and filed the following: http://jira.codehaus.org/browse/JRUBY-5410 I want us to publish a *GPL-free maven artifact for JRuby 1.6.0. This should be usable by Apache and other projects that can't include GPLed code. I am talking to the maintainer of jffi (and jaffl, which was incorrectly listed as MIT licensed in COPYING) about adding an Apache-friendly license. I added a jar-jruby-nogpl target to our build.xml and fixed COPYING to properly reflect jaffl licensing and jffi and jaffl homepages I also confirmed that simply removing jffi, jaffl, and jgrapht from jruby.jar does not damage general Ruby behavior. As mentioned above, some native or system-level capabilities are disabled or work differently, but IRB, RubyGems, and Rails all function correctly.
        Hide
        Charles Oliver Nutter added a comment -

        The author of jffi and jaffl has asked if Sun's CDDL would be an acceptable license under which he could dual-license jffi, jaffl, and other projects. Is CDDL acceptable?

        Show
        Charles Oliver Nutter added a comment - The author of jffi and jaffl has asked if Sun's CDDL would be an acceptable license under which he could dual-license jffi, jaffl, and other projects. Is CDDL acceptable?
        Hide
        Charles Oliver Nutter added a comment -

        My reading of Apache's position on CDDL (from http://www.apache.org/legal/3party.html#criteriaandcategories) is that CDDL would be acceptable if the software/library in question is distributed only in binary form and an appropriate note of the so-licensed software is made in the LICENSE file. This appears to puts CDDL in the same category as CPL, under which license JRuby itself is allowed for inclusion in HBase.

        So I think CDDL would be acceptable...correct?

        Show
        Charles Oliver Nutter added a comment - My reading of Apache's position on CDDL (from http://www.apache.org/legal/3party.html#criteriaandcategories ) is that CDDL would be acceptable if the software/library in question is distributed only in binary form and an appropriate note of the so-licensed software is made in the LICENSE file. This appears to puts CDDL in the same category as CPL, under which license JRuby itself is allowed for inclusion in HBase. So I think CDDL would be acceptable...correct?
        Hide
        Andrew Purtell added a comment -

        Alright Charles, leaving the meta issue of whether Ruby syntax is really what people want long term for the shell, reopening this issue perhaps to upgrade to an ASF-clean JRuby for the next HBase release if JRuby sorts the licensing issues out.

        Show
        Andrew Purtell added a comment - Alright Charles, leaving the meta issue of whether Ruby syntax is really what people want long term for the shell, reopening this issue perhaps to upgrade to an ASF-clean JRuby for the next HBase release if JRuby sorts the licensing issues out.
        Andrew Purtell made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Assignee stack [ stack ]
        Andrew Purtell made changes -
        Fix Version/s 0.90.1 [ 12315548 ]
        Fix Version/s 0.92.0 [ 12314223 ]
        Fix Version/s 0.90.0 [ 12313607 ]
        Priority Blocker [ 1 ] Major [ 3 ]
        Hide
        stack added a comment -

        @Charles Yes, CDDL looks fine. We can add appropriate notices np. Thanks for taking this on.

        Show
        stack added a comment - @Charles Yes, CDDL looks fine. We can add appropriate notices np. Thanks for taking this on.
        Hide
        Charles Oliver Nutter added a comment -

        Andrew, stack: Thank you for giving us an opportunity to fix this

        An update for you.

        Due to the oldness of CDDL, the author of jffi and jaffl (Wayne Meissner, also one of the primary – or the primary – contributors to jna) has opted to add EPL as an additional license. This is roughly in line with our CPL license in JRuby and falls into the same category as CPL and CDDL as far as inclusion in Apache projects.

        We will also strip jgrapht from JRuby 1.6.0.RC2, so the binaries for JRuby 1.6 final will not have any GPL or LGPL-only library dependencies.

        I'm very glad we were able to get this sorted out.

        Show
        Charles Oliver Nutter added a comment - Andrew, stack: Thank you for giving us an opportunity to fix this An update for you. Due to the oldness of CDDL, the author of jffi and jaffl (Wayne Meissner, also one of the primary – or the primary – contributors to jna) has opted to add EPL as an additional license. This is roughly in line with our CPL license in JRuby and falls into the same category as CPL and CDDL as far as inclusion in Apache projects. We will also strip jgrapht from JRuby 1.6.0.RC2, so the binaries for JRuby 1.6 final will not have any GPL or LGPL-only library dependencies. I'm very glad we were able to get this sorted out.
        Hide
        stack added a comment -

        @Charles sweet!

        Show
        stack added a comment - @Charles sweet!
        Hide
        Andrew Purtell added a comment -

        @Charles Thanks, looking forward to JRuby 1.6.

        Show
        Andrew Purtell added a comment - @Charles Thanks, looking forward to JRuby 1.6.
        Hide
        Charles Oliver Nutter added a comment -

        A final update, and then hopefully this will be all set for JRuby 1.6RC2:

        Wayne has decided to go ahead and just add Apache-2.0 to his jffi and jaffl projects. I guess that should settle things nicely

        Show
        Charles Oliver Nutter added a comment - A final update, and then hopefully this will be all set for JRuby 1.6RC2: Wayne has decided to go ahead and just add Apache-2.0 to his jffi and jaffl projects. I guess that should settle things nicely
        Hide
        stack added a comment -

        @Charles That'll work (smile).

        Show
        stack added a comment - @Charles That'll work (smile).
        Hide
        Stephen Watt added a comment -

        @Charles Thanks for the contributions. This is some awesome cross-project synergy.

        @stack et al - I'll do another legal scan of HBase for 0.90.1 once its out and will let you know if I find any other issues. Thanks for resolving this so quickly.

        Show
        Stephen Watt added a comment - @Charles Thanks for the contributions. This is some awesome cross-project synergy. @stack et al - I'll do another legal scan of HBase for 0.90.1 once its out and will let you know if I find any other issues. Thanks for resolving this so quickly.
        stack made changes -
        Release Note The lads over in jruby land say 1.6RC2 should have the licensing fixup. Lets get it into TRUNK when ready. I don't think this a 0.90.x issue. Moving it out.
        Fix Version/s 0.90.1 [ 12315548 ]
        Hide
        Charles Oliver Nutter added a comment -

        The JRuby bug (JRUBY-5410) has been resolved! JRuby 1.6.0.RC2 will ship with no [L]GPL-only libraries, now that jaffl, jffi, and jnr-netdb are licensed Apache-2.0 and jgrapht has been removed from our binary distribution.

        Thanks for your patience!

        Show
        Charles Oliver Nutter added a comment - The JRuby bug (JRUBY-5410) has been resolved! JRuby 1.6.0.RC2 will ship with no [L] GPL-only libraries, now that jaffl, jffi, and jnr-netdb are licensed Apache-2.0 and jgrapht has been removed from our binary distribution. Thanks for your patience!
        Hide
        stack added a comment -

        Thanks @Charles. We'll wait on RC2 release then bring it in and close this issue. Good stuff.

        Show
        stack added a comment - Thanks @Charles. We'll wait on RC2 release then bring it in and close this issue. Good stuff.
        Hide
        stack added a comment -

        Closing. Opened HBASE-3600 to accomodate update to jruby 1.6.0RC2 (Charles, if you see this, I notice that 1.6.0RC2 is out but its not in a maven repo – you know why? Usually RCs make it to maven it would seem. Thanks, and thanks again for fixing licensing so we could use the new stuff again)

        Show
        stack added a comment - Closing. Opened HBASE-3600 to accomodate update to jruby 1.6.0RC2 (Charles, if you see this, I notice that 1.6.0RC2 is out but its not in a maven repo – you know why? Usually RCs make it to maven it would seem. Thanks, and thanks again for fixing licensing so we could use the new stuff again)
        Hide
        stack added a comment -

        Actually closing.

        Show
        stack added a comment - Actually closing.
        stack made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Assignee stack [ stack ]
        Resolution Fixed [ 1 ]

          People

          • Assignee:
            stack
            Reporter:
            stack
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development