Lucene - Core
  1. Lucene - Core
  2. LUCENE-3167

Make lucene/solr a OSGI bundle through Ant

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      bndtools

    • Lucene Fields:
      New

      Description

      We need to make a bundle thriugh Ant, so the binary can be published and no more need the download of the sources. Actually to get a OSGI bundle we need to use maven tools and build the sources. Here the reference for the creation of the OSGI bundle through Maven:

      https://issues.apache.org/jira/browse/LUCENE-1344

      Bndtools could be used inside Ant

      1. lucene_trunk.patch
        6 kB
        Luca Stancapiano
      2. lucene_trunk.patch
        6 kB
        Luca Stancapiano
      3. LUCENE-3167.patch
        9 kB
        Luca Stancapiano
      4. LUCENE-3167.patch
        32 kB
        Steve Rowe
      5. LUCENE-3167.patch
        46 kB
        Luca Stancapiano
      6. LUCENE-3167_20130108.patch
        45 kB
        Tommaso Teofili

        Issue Links

          Activity

          Hide
          Nicolas Lalevée added a comment -

          As wrote earlier, Ivy is indeed useless to build an OSGi Manifest, it is "just" able to understand an OSGi remote repository. Here Ant + bndTools is the perfect tool combination for this. (BTW I'm a maintainer of Ivy).

          Show
          Nicolas Lalevée added a comment - As wrote earlier, Ivy is indeed useless to build an OSGi Manifest, it is "just" able to understand an OSGi remote repository. Here Ant + bndTools is the perfect tool combination for this. (BTW I'm a maintainer of Ivy).
          Hide
          Tommaso Teofili added a comment -

          from a first inspection it doesn't seem to enhance the basic Ant + bndTools based approach too much therefore I'm not sure it's worth going that path.

          Show
          Tommaso Teofili added a comment - from a first inspection it doesn't seem to enhance the basic Ant + bndTools based approach too much therefore I'm not sure it's worth going that path.
          Hide
          Tommaso Teofili added a comment -

          reading again the whole history, and especially Robert's comment on eventually leveraging Ivy for this, I found https://ant.apache.org/ivy/history/trunk/osgi.html

          I don't know if and how it could be useful, I'll have a look and in case let you know.

          Show
          Tommaso Teofili added a comment - reading again the whole history, and especially Robert's comment on eventually leveraging Ivy for this, I found https://ant.apache.org/ivy/history/trunk/osgi.html I don't know if and how it could be useful, I'll have a look and in case let you know.
          Hide
          Robert Muir added a comment -

          Maybe if the Lucene committers doesn't feel enough confortable with OSGi, they should let it to some external OSGi packagers, just like there are debian packagers. Here is a exemple of an Ivy repository maintained by 'packagers': https://code.google.com/p/ivyroundup/. There was a tentative for OSGi but it stalled: https://github.com/glyn/bundlerepo

          Yes, I think someone downstream should do it, outside of this project.

          I said the same thing about maven, that one didnt work out. But this had to do more with maven advocates lying about the necessity of it taking place in this project: turned out later this wasn't true and pretty much anybody can release anybody else's shit on maven central.

          I won't make the same mistake twice.

          Show
          Robert Muir added a comment - Maybe if the Lucene committers doesn't feel enough confortable with OSGi, they should let it to some external OSGi packagers, just like there are debian packagers. Here is a exemple of an Ivy repository maintained by 'packagers': https://code.google.com/p/ivyroundup/ . There was a tentative for OSGi but it stalled: https://github.com/glyn/bundlerepo Yes, I think someone downstream should do it, outside of this project. I said the same thing about maven, that one didnt work out. But this had to do more with maven advocates lying about the necessity of it taking place in this project: turned out later this wasn't true and pretty much anybody can release anybody else's shit on maven central. I won't make the same mistake twice.
          Hide
          Nicolas Lalevée added a comment -

          If I simplify, Maven dependency management is about declaring that a jar depends on another. OSGi is more powerful/complex than that. It can do that jar to jar dependency, but it is not recommended by the advocates of OSGi. The prefered way is when dependencies are declared between the java packages. So instead of declaring a dependency on a jar, you declare a dependency on a java package. It is then like declaring a dependency on an API rather than an implementation. And then comes versioning into the mix : you have a version of an API (the java packages) and the version of the implementation (the jar/bundle). So that static metadata in the manifest is not trivial to maintain.
          Then here comes some tools to help, the patches here use "bnd". The java packages are somewhat part of the Java type system, so tools like bnd you can infer nearly automatically these dependencies.

          Also note that contrary to the Maven world, the OSGi world doesn't have a world class reference repository. So you do not need to upload it anywhere else. I guess this issue is "just" about making the Lucene jars droppable in the OSGi environement as is.

          Maybe if the Lucene committers doesn't feel enough confortable with OSGi, they should let it to some external OSGi packagers, just like there are debian packagers. Here is a exemple of an Ivy repository maintained by 'packagers': https://code.google.com/p/ivyroundup/. There was a tentative for OSGi but it stalled: https://github.com/glyn/bundlerepo

          Show
          Nicolas Lalevée added a comment - If I simplify, Maven dependency management is about declaring that a jar depends on another. OSGi is more powerful/complex than that. It can do that jar to jar dependency, but it is not recommended by the advocates of OSGi. The prefered way is when dependencies are declared between the java packages. So instead of declaring a dependency on a jar, you declare a dependency on a java package. It is then like declaring a dependency on an API rather than an implementation. And then comes versioning into the mix : you have a version of an API (the java packages) and the version of the implementation (the jar/bundle). So that static metadata in the manifest is not trivial to maintain. Then here comes some tools to help, the patches here use "bnd". The java packages are somewhat part of the Java type system, so tools like bnd you can infer nearly automatically these dependencies. Also note that contrary to the Maven world, the OSGi world doesn't have a world class reference repository. So you do not need to upload it anywhere else. I guess this issue is "just" about making the Lucene jars droppable in the OSGi environement as is. Maybe if the Lucene committers doesn't feel enough confortable with OSGi, they should let it to some external OSGi packagers, just like there are debian packagers. Here is a exemple of an Ivy repository maintained by 'packagers': https://code.google.com/p/ivyroundup/ . There was a tentative for OSGi but it stalled: https://github.com/glyn/bundlerepo
          Hide
          Tommaso Teofili added a comment -

          The official Lucene Maven artifacts as uploaded to Maven Central are not built by Maven! They are build by ANT

          that's what I knew, but I then misunderstood your sentence when you said "we release with Maven", sorry for the confusion

          This means the missing meta-inf must be added by ANT. The question I have: Why no simple change the JAR task and add the few additional key-value pairs? This would be a simple patch, if it is just some more-or-less static properties derived from some ANT build properties (like module name, version,...).

          good point

          Show
          Tommaso Teofili added a comment - The official Lucene Maven artifacts as uploaded to Maven Central are not built by Maven! They are build by ANT that's what I knew, but I then misunderstood your sentence when you said "we release with Maven", sorry for the confusion This means the missing meta-inf must be added by ANT. The question I have: Why no simple change the JAR task and add the few additional key-value pairs? This would be a simple patch, if it is just some more-or-less static properties derived from some ANT build properties (like module name, version,...). good point
          Hide
          Uwe Schindler added a comment -

          The official Lucene Maven artifacts as uploaded to Maven Central are not built by Maven! They are build by ANT. The POM files in dev-tools are a separate build environment thats not used for releasing.

          This means the missing meta-inf must be added by ANT. The question I have: Why no simple change the JAR task and add the few additional key-value pairs? This would be a simple patch, if it is just some more-or-less static properties derived from some ANT build properties (like module name, version,...).

          Show
          Uwe Schindler added a comment - The official Lucene Maven artifacts as uploaded to Maven Central are not built by Maven! They are build by ANT. The POM files in dev-tools are a separate build environment thats not used for releasing. This means the missing meta-inf must be added by ANT. The question I have: Why no simple change the JAR task and add the few additional key-value pairs? This would be a simple patch, if it is just some more-or-less static properties derived from some ANT build properties (like module name, version,...).
          Hide
          Tommaso Teofili added a comment -

          My last comment: If it is only adding some static strings to the META-INF of every JAR, who cares? But if you need crazy magic or must upload the JAR files to another site on the net, I am also -1!

          basically I do agree with the fact that we should keep the release process as smooth as possible, that's why I was not pushing too much this myself.
          I don't think we should upload artifacts anywhere else (and I don't think that's the point here), using the BND Ant tasks with our Ant based build is instead not really smooth, at least as far as I could do.
          My personal point here is if the Maven build already provides an "OSGi friendly" jar / manifest then (to me) we can mark this as resolved as I don't see why we should duplicate our efforts; probably we should instead fix the Maven configuration for the OSGi manifest as till now when I tried to deploy a released Lucene/Solr package into an OSGi container it didn't work correctly as the required manifest information wasn't there (which leads me to think that the problem may be related to the Maven profile used for the release).

          My 2 cents.

          Show
          Tommaso Teofili added a comment - My last comment: If it is only adding some static strings to the META-INF of every JAR, who cares? But if you need crazy magic or must upload the JAR files to another site on the net, I am also -1! basically I do agree with the fact that we should keep the release process as smooth as possible, that's why I was not pushing too much this myself. I don't think we should upload artifacts anywhere else (and I don't think that's the point here), using the BND Ant tasks with our Ant based build is instead not really smooth, at least as far as I could do. My personal point here is if the Maven build already provides an "OSGi friendly" jar / manifest then (to me) we can mark this as resolved as I don't see why we should duplicate our efforts; probably we should instead fix the Maven configuration for the OSGi manifest as till now when I tried to deploy a released Lucene/Solr package into an OSGi container it didn't work correctly as the required manifest information wasn't there (which leads me to think that the problem may be related to the Maven profile used for the release). My 2 cents.
          Hide
          Uwe Schindler added a comment -

          Just a very stupid question: Looking for "OSGI" in Wikipedia and other pages does not make it more clear to me. It confuses me more - this looks to me like completely useless and a duplicate of what Maven provides. Why do we need it when we release with Maven? Just because the Eclipse Foundation does not like Maven and wants to use their own "standard" OSGI? We decided to use Maven for publishing artifacts (in addition to the source and binary TGZ files). In my opinion, the Eclipse Foundation should provide a "converter" that makes Maven artifacts available to OSGI, they cannot force every project to support just another framework.

          My last comment: If it is only adding some static strings to the META-INF of every JAR, who cares? But if you need crazy magic or must upload the JAR files to another site on the net, I am also -1!

          Show
          Uwe Schindler added a comment - Just a very stupid question: Looking for "OSGI" in Wikipedia and other pages does not make it more clear to me. It confuses me more - this looks to me like completely useless and a duplicate of what Maven provides. Why do we need it when we release with Maven? Just because the Eclipse Foundation does not like Maven and wants to use their own "standard" OSGI? We decided to use Maven for publishing artifacts (in addition to the source and binary TGZ files). In my opinion, the Eclipse Foundation should provide a "converter" that makes Maven artifacts available to OSGI, they cannot force every project to support just another framework. My last comment: If it is only adding some static strings to the META-INF of every JAR, who cares? But if you need crazy magic or must upload the JAR files to another site on the net, I am also -1!
          Hide
          Robert Muir added a comment -

          Bottom line? I'd rather you guys do it – even if you don't do a good job of it. Even if I could do it 10x better, it is still 100x better for you to do it.

          Sure, its always better to ask someone else to take the burden. But I want the release process to continue to be smooth. The less unnecessary things we take on the better. The less manual stuff that needs to be checked the better.

          We could package up linux RPMs and freebsd ports too, but we just leave this for downstream people. Could we do it better than those folks since we know the packaging? I don't know, but I don't really care either. Its better that we spend our time working on search engines so there are actual releases to put out there.

          Recently I threw up a 4.2.0 release candidate on a wednesday night, by monday the release was out. By wednesday my friend forwards an email from freebsd showing they had updated their package already: http://svnweb.freebsd.org/ports?view=revision&revision=314108 . Thats pretty damn efficient downstream packaging and a relatively short release cycle. It required almost no manual effort: I'm insisting that we keep it this way.

          So again, this is just like maven. I push back hard, really hard on the issue because I'm really just saying "don't add a feature without tests and force the release manager to manually verify the shit works". I said the same things about maven and someone (thanks Steve!) added automatic testing of maven and then I never said another word about it.

          Show
          Robert Muir added a comment - Bottom line? I'd rather you guys do it – even if you don't do a good job of it. Even if I could do it 10x better, it is still 100x better for you to do it. Sure, its always better to ask someone else to take the burden. But I want the release process to continue to be smooth. The less unnecessary things we take on the better. The less manual stuff that needs to be checked the better. We could package up linux RPMs and freebsd ports too, but we just leave this for downstream people. Could we do it better than those folks since we know the packaging? I don't know, but I don't really care either. Its better that we spend our time working on search engines so there are actual releases to put out there. Recently I threw up a 4.2.0 release candidate on a wednesday night, by monday the release was out. By wednesday my friend forwards an email from freebsd showing they had updated their package already: http://svnweb.freebsd.org/ports?view=revision&revision=314108 . Thats pretty damn efficient downstream packaging and a relatively short release cycle. It required almost no manual effort: I'm insisting that we keep it this way. So again, this is just like maven. I push back hard, really hard on the issue because I'm really just saying "don't add a feature without tests and force the release manager to manually verify the shit works". I said the same things about maven and someone (thanks Steve!) added automatic testing of maven and then I never said another word about it.
          Hide
          Bob Kerns added a comment - - edited

          To respond to Robert Muir's question about why you (people who work on search engines) should do it.

          I see two reasons:

          1) People downstream may not KNOW how to do it, or have the time, or want to make all the possible choices over again, when people who know the engine and its packaging are more capable of sorting all that stuff out properly. I didn't get it right the first time (thank's to screwing up with) and now have to repeat. It's taken you guys a year and a half; obviously not trivial!

          2) People downstream would like to do it in a way that does not outright conflict with other people downstream doing it! This is really a big, big deal. If I package it, and put it in a product, and someone else packages it, and puts it in their product – like for example, the Eclipse foundation – bad things can happen. I can't even predict WHAT bad things might happen, because I don't know what we might do differently. We could get lucky and not screw each other, but do I really want to trust the other guy – ALL the other guys – to do it right? Or do I do something like rename the packages to avoid any potential insanity? That has other problems...

          Bottom line? I'd rather you guys do it – even if you don't do a good job of it. Even if I could do it 10x better, it is still 100x better for you to do it.

          Maybe I'll be able to help later, once I've sorted out my immediate needs. But in the meantime, I just wanted to let you know the effort is appreciated – even if it doesn't turn out perfectly.

          Thanks.

          Show
          Bob Kerns added a comment - - edited To respond to Robert Muir's question about why you (people who work on search engines) should do it. I see two reasons: 1) People downstream may not KNOW how to do it, or have the time, or want to make all the possible choices over again, when people who know the engine and its packaging are more capable of sorting all that stuff out properly. I didn't get it right the first time (thank's to screwing up with) and now have to repeat. It's taken you guys a year and a half; obviously not trivial! 2) People downstream would like to do it in a way that does not outright conflict with other people downstream doing it! This is really a big, big deal. If I package it, and put it in a product, and someone else packages it, and puts it in their product – like for example, the Eclipse foundation – bad things can happen. I can't even predict WHAT bad things might happen, because I don't know what we might do differently. We could get lucky and not screw each other, but do I really want to trust the other guy – ALL the other guys – to do it right? Or do I do something like rename the packages to avoid any potential insanity? That has other problems... Bottom line? I'd rather you guys do it – even if you don't do a good job of it. Even if I could do it 10x better, it is still 100x better for you to do it. Maybe I'll be able to help later, once I've sorted out my immediate needs. But in the meantime, I just wanted to let you know the effort is appreciated – even if it doesn't turn out perfectly. Thanks.
          Hide
          Tommaso Teofili added a comment - - edited

          New not final version of the patch.

          I'm not an Ant expert so I may be missing something, if that's the case feel free to correct / point me to where / how fix things.

          With this new one it's possible to run ant compile -Dosgi=true to have OSGi ready jars' manifests so that single modules can be installed and started in an OSGi container.
          By default however the OSGi build doesn't happen.

          What is still missing is a smart and non intrusive way of setting correctly the OSGi exported-package directive. Currently that's set to: org.apache.lucene*;version=$

          {version},org.apache.lucene.*;version=${version}

          ;-split-package:=merge-first for all the bundles which is not correct.

          As a side note I noticed that in lucene/core/build.xml the classpath is not set while is set to the core files in the root build.xml; this seems to me a bit odd, is there any specific reason for that?
          In the patch I temporarily added the correct classpath in lucene/core/build.xml (which is needed for the OSGi build for the core to export the packages).

          Show
          Tommaso Teofili added a comment - - edited New not final version of the patch. I'm not an Ant expert so I may be missing something, if that's the case feel free to correct / point me to where / how fix things. With this new one it's possible to run ant compile -Dosgi=true to have OSGi ready jars' manifests so that single modules can be installed and started in an OSGi container. By default however the OSGi build doesn't happen. What is still missing is a smart and non intrusive way of setting correctly the OSGi exported-package directive. Currently that's set to: org.apache.lucene*;version=$ {version},org.apache.lucene.*;version=${version} ;-split-package:=merge-first for all the bundles which is not correct. As a side note I noticed that in lucene/core/build.xml the classpath is not set while is set to the core files in the root build.xml; this seems to me a bit odd, is there any specific reason for that? In the patch I temporarily added the correct classpath in lucene/core/build.xml (which is needed for the OSGi build for the core to export the packages).
          Hide
          Luca Stancapiano added a comment -

          great, daje Tom

          Show
          Luca Stancapiano added a comment - great, daje Tom
          Hide
          Tommaso Teofili added a comment -

          now trying to rework Luca's patch to see if we can improve it, and have the OSGi manifest build disabled by default but maybe enabled for release and on demand (e.g. -Dosgi=true).

          Show
          Tommaso Teofili added a comment - now trying to rework Luca's patch to see if we can improve it, and have the OSGi manifest build disabled by default but maybe enabled for release and on demand (e.g. -Dosgi=true).
          Hide
          Robert Muir added a comment -

          What Gunnar Wagenknecht says he's doing now seems close?:

          Right, which begs the question if people who understand this stuff can already
          do this downstream, why should we (people who work on search engines) do it.

          Its just more stuff to go wrong when trying to release.

          Show
          Robert Muir added a comment - What Gunnar Wagenknecht says he's doing now seems close?: Right, which begs the question if people who understand this stuff can already do this downstream, why should we (people who work on search engines) do it. Its just more stuff to go wrong when trying to release.
          Hide
          Tommaso Teofili added a comment -

          In my opinion doing and (especially) maintaining the OSGi packages from ant would be a pain, on the other hand I can see that keeping two different versioning could be hard as well. That's the reason way I thought that, at least to start (as Jukka told), a facade package for explicitly exporting the main stuff for OSGi could be a solution.
          There someone with enough OSGi experience could take care of the whole thing without causing much impact on the project.
          What I personally look for is having something that works out without much pain and Jukka's proposal looks to me to go in that direction.
          Obviously if there are other ideas I think that they'd be more than welcome.
          Gunnar's proposal could be ok as a starting point as well, but actually I don't know how hard it'd be to do that automatically with our build mechanism.

          Show
          Tommaso Teofili added a comment - In my opinion doing and (especially) maintaining the OSGi packages from ant would be a pain, on the other hand I can see that keeping two different versioning could be hard as well. That's the reason way I thought that, at least to start (as Jukka told), a facade package for explicitly exporting the main stuff for OSGi could be a solution. There someone with enough OSGi experience could take care of the whole thing without causing much impact on the project. What I personally look for is having something that works out without much pain and Jukka's proposal looks to me to go in that direction. Obviously if there are other ideas I think that they'd be more than welcome. Gunnar's proposal could be ok as a starting point as well, but actually I don't know how hard it'd be to do that automatically with our build mechanism.
          Hide
          Robert Muir added a comment -

          Then we are back to my original suggestion above, that we start with just the ant integration
          as maven was done and don't try to change versioning nor back compat (this will be controversial,
          at least I will become really really annoying).

          Show
          Robert Muir added a comment - Then we are back to my original suggestion above, that we start with just the ant integration as maven was done and don't try to change versioning nor back compat (this will be controversial, at least I will become really really annoying).
          Hide
          Steve Rowe added a comment -

          I disagree with the whitepaper too, except the part where is says that semantic versioning is optional.

          Show
          Steve Rowe added a comment - I disagree with the whitepaper too, except the part where is says that semantic versioning is optional.
          Hide
          Robert Muir added a comment -

          This conflicts with that PDF though... I guess I'm saying I disagree with that PDF completely

          Show
          Robert Muir added a comment - This conflicts with that PDF though... I guess I'm saying I disagree with that PDF completely
          Hide
          Steve Rowe added a comment -

          Robert Muir wrote:

          A better approach would be to adapt the OSGI integration to work well with Lucene's existing backwards compatibility policy (http://wiki.apache.org/lucene-java/BackwardsCompatibility) instead of the other way around.

          What Gunnar Wagenknecht says he's doing now seems close?:

          Having real semantic versioning at the package level would be good for OSGi. However, for the time being, I simply use the main Lucene version for all package exports. That works today and should also be a first step here.

          Show
          Steve Rowe added a comment - Robert Muir wrote: A better approach would be to adapt the OSGI integration to work well with Lucene's existing backwards compatibility policy ( http://wiki.apache.org/lucene-java/BackwardsCompatibility ) instead of the other way around. What Gunnar Wagenknecht says he's doing now seems close?: Having real semantic versioning at the package level would be good for OSGi. However, for the time being, I simply use the main Lucene version for all package exports. That works today and should also be a first step here.
          Hide
          Steve Rowe added a comment -

          I count 280 packages in Lucene/Solr.

          Show
          Steve Rowe added a comment - I count 280 packages in Lucene/Solr.
          Hide
          Robert Muir added a comment -

          Wait: I'm saying I'm not sure these additional backwards compat things should be done at all.

          Changing things about the backwards compatibility policy to make development and releasing
          more difficult is serious and I'm not seeing the benefit here.

          A better approach would be to adapt the OSGI integration to work well with Lucene's
          existing backwards compatibility policy (http://wiki.apache.org/lucene-java/BackwardsCompatibility)
          instead of the other way around.

          Show
          Robert Muir added a comment - Wait: I'm saying I'm not sure these additional backwards compat things should be done at all. Changing things about the backwards compatibility policy to make development and releasing more difficult is serious and I'm not seeing the benefit here. A better approach would be to adapt the OSGI integration to work well with Lucene's existing backwards compatibility policy ( http://wiki.apache.org/lucene-java/BackwardsCompatibility ) instead of the other way around.
          Hide
          Gunnar Wagenknecht added a comment -

          Jukka, these are all good point. I'm the maintainer of the Lucene OSGi bundles in Eclipse with quite a bit experience in OSGi-ifying libraries. There is a trade-off between satisfying all of the points and being pragmatic.

          Having real semantic versioning at the package level would be good for OSGi. However, for the time being, I simply use the main Lucene version for all package exports. That works today and should also be a first step here.

          Future work (like removing split packages and package level versioning) should be done as separate steps.

          Show
          Gunnar Wagenknecht added a comment - Jukka, these are all good point. I'm the maintainer of the Lucene OSGi bundles in Eclipse with quite a bit experience in OSGi-ifying libraries. There is a trade-off between satisfying all of the points and being pragmatic. Having real semantic versioning at the package level would be good for OSGi. However, for the time being, I simply use the main Lucene version for all package exports. That works today and should also be a first step here. Future work (like removing split packages and package level versioning) should be done as separate steps.
          Hide
          Robert Muir added a comment -

          This seems like a significantly more complicated issue then than just modifying the ant build to add metadata.
          Maybe this build step should be addressed first (so someone can optionally do it with ant if they want), just like maven
          and leave the thornier back compat issue for another issue.

          From this PDF it seems OSGI imposes its own backwards compatibility policy that conflicts with ours, and would
          be very difficult to maintain and detect if they were correct.

          I don't see why this would be useful at all: in basically every release binary compatibility is usually
          broken in some way (and even if its not technically broken, maybe a methods behavior changes or something
          like). Different lucene modules depend on each other and are only tested with the same version: today
          its not tested to use lucene-foobar-3.2.jar with a lucene-core-3.1.jar.

          So the versioning in that whitepaper doesn't make much sense: I think if we are going to put OSGI information
          in the packages we should simply bump the major version for every release: even bugfix releases.

          Show
          Robert Muir added a comment - This seems like a significantly more complicated issue then than just modifying the ant build to add metadata. Maybe this build step should be addressed first (so someone can optionally do it with ant if they want), just like maven and leave the thornier back compat issue for another issue. From this PDF it seems OSGI imposes its own backwards compatibility policy that conflicts with ours, and would be very difficult to maintain and detect if they were correct. I don't see why this would be useful at all: in basically every release binary compatibility is usually broken in some way (and even if its not technically broken, maybe a methods behavior changes or something like). Different lucene modules depend on each other and are only tested with the same version: today its not tested to use lucene-foobar-3.2.jar with a lucene-core-3.1.jar. So the versioning in that whitepaper doesn't make much sense: I think if we are going to put OSGI information in the packages we should simply bump the major version for every release: even bugfix releases.
          Hide
          Jukka Zitting added a comment -

          We discussed this briefly with Tommaso and a few OSGi experts last week.

          While adding bundle metadata to the Lucene jars is fairly straightforward to implement, doing so comes with a commitment to maintain correct semantic versioning at the package-level (see http://www.osgi.org/wiki/uploads/Links/SemanticVersioning.pdf). Without such a commitment the package version information will eventually become outdated and incorrect, which can lead to tricky problems for OSGi deployments. Most notably properly managed package versions will often not follow the main product version numbers, as even when backwards-incompatible changes is introduced in one package, another package might still remain unchanged.

          The best incremental solution here could be to initially export only one or two key packages for which semantic version information can be maintained without too much overhead going forward. A good starting point could be something like a separate facade package for which stronger-than-usual backwards compatibility promises can be made. Once there's more OSGi experience in the core Lucene community, more packages can be exported to OSGi clients.

          Show
          Jukka Zitting added a comment - We discussed this briefly with Tommaso and a few OSGi experts last week. While adding bundle metadata to the Lucene jars is fairly straightforward to implement, doing so comes with a commitment to maintain correct semantic versioning at the package-level (see http://www.osgi.org/wiki/uploads/Links/SemanticVersioning.pdf ). Without such a commitment the package version information will eventually become outdated and incorrect, which can lead to tricky problems for OSGi deployments. Most notably properly managed package versions will often not follow the main product version numbers, as even when backwards-incompatible changes is introduced in one package, another package might still remain unchanged. The best incremental solution here could be to initially export only one or two key packages for which semantic version information can be maintained without too much overhead going forward. A good starting point could be something like a separate facade package for which stronger-than-usual backwards compatibility promises can be made. Once there's more OSGi experience in the core Lucene community, more packages can be exported to OSGi clients.
          Hide
          Luca Stancapiano added a comment -

          Hi guys, Nicolas, I confirm for the October 2011... in that time the patch worked.....I'm wondered that none has still committed the work...I will be here for help. Let me know!

          Show
          Luca Stancapiano added a comment - Hi guys, Nicolas, I confirm for the October 2011... in that time the patch worked.....I'm wondered that none has still committed the work...I will be here for help. Let me know!
          Hide
          Tommaso Teofili added a comment -

          sure, and I'd be happy to lend a hand here.

          Show
          Tommaso Teofili added a comment - sure, and I'd be happy to lend a hand here.
          Hide
          Michael McCandless added a comment -

          Hi Jukka, I don't think anyone is working on this (we don't really collectively have enough OSGI expertise around...). So if you can help out that would be awesome! Thanks.

          Show
          Michael McCandless added a comment - Hi Jukka, I don't think anyone is working on this (we don't really collectively have enough OSGI expertise around...). So if you can help out that would be awesome! Thanks.
          Hide
          Jukka Zitting added a comment -

          Is anyone actively working on this (i.e. is Luca's patch from October 2011 the latest one)? If not, I can take a look at addressing the concerns raised above.

          Show
          Jukka Zitting added a comment - Is anyone actively working on this (i.e. is Luca's patch from October 2011 the latest one)? If not, I can take a look at addressing the concerns raised above.
          Hide
          Nicolas Lalevée added a comment -

          Ivy won't help you for this issue which is about adding OSGi metadata into the jar. Ivy can understand OSGi manifests but it is unable to write them. Tools like bnd (used in the patch IIRC) is definitively the way to go.

          Show
          Nicolas Lalevée added a comment - Ivy won't help you for this issue which is about adding OSGi metadata into the jar. Ivy can understand OSGi manifests but it is unable to write them. Tools like bnd (used in the patch IIRC) is definitively the way to go.
          Hide
          Robert Muir added a comment -

          Out of curiousity, now that we are using ivy in our ant build, has anyone investigated
          the upcoming ivy support for OSGI?

          Would it do what we want? http://ant.apache.org/ivy/history/trunk/osgi.html, http://ant.apache.org/ivy/history/trunk/osgi/standard-osgi.html, etc.

          It seems to use the same utilities here, so I have a few concerns:

          • it if slows down the build 90seconds, I think its better that to support OSGI this
            would only be done inside prepare-release, not during normal 'ant test'. This way
            our ordinary compile-test-debug cycle is unaffected. This is really important.
          • I don't know anything about OSGI, but how do we know its "correct"? Just like any
            thing else we include in our release, we shouldnt just be releasing arbitrary stuff
            (even additional metadata) unless we have some mechanism to confirm that its correct.
          • the current ivy integration is labeled as experimental (making the above even more important).

          anyway I know nothing of OSGI, so maybe this is useless, I just wanted to bring it up.

          Show
          Robert Muir added a comment - Out of curiousity, now that we are using ivy in our ant build, has anyone investigated the upcoming ivy support for OSGI? Would it do what we want? http://ant.apache.org/ivy/history/trunk/osgi.html , http://ant.apache.org/ivy/history/trunk/osgi/standard-osgi.html , etc. It seems to use the same utilities here, so I have a few concerns: it if slows down the build 90seconds, I think its better that to support OSGI this would only be done inside prepare-release, not during normal 'ant test'. This way our ordinary compile-test-debug cycle is unaffected. This is really important. I don't know anything about OSGI, but how do we know its "correct"? Just like any thing else we include in our release, we shouldnt just be releasing arbitrary stuff (even additional metadata) unless we have some mechanism to confirm that its correct. the current ivy integration is labeled as experimental (making the above even more important). anyway I know nothing of OSGI, so maybe this is useless, I just wanted to bring it up.
          Hide
          Ryan Hill added a comment -

          Thanks for the helpful digest. Your approach sounds completely reasonable to me.

          I don't fiddle with ant much, but I'll take a look.

          Show
          Ryan Hill added a comment - Thanks for the helpful digest. Your approach sounds completely reasonable to me. I don't fiddle with ant much, but I'll take a look.
          Hide
          Steve Rowe added a comment -

          The way forward, as I see it:

          1. Make a new property to control building of OSGi manifests, and give it a properly descriptive name, e.g. "build-osgi-manifests" (unlike in the latest patch, where "dev" is used as the property name, AFAICT to intentionally obfuscate the fact that OSGi manifests are being built).
          2. Make the default "build-osgi-manifests" property value false, so that devs don't have to wait around the extra 90 seconds on every build (unlike in the latest patch, where the default value is true).
          3. Include solr/ and modules/ in the build modifications (unlike the latest patch, which excludes them).
          4. Cause the "build-osgi-manifests" property to be set to true when packaging releases.
          Show
          Steve Rowe added a comment - The way forward, as I see it: Make a new property to control building of OSGi manifests, and give it a properly descriptive name, e.g. "build-osgi-manifests" (unlike in the latest patch, where "dev" is used as the property name, AFAICT to intentionally obfuscate the fact that OSGi manifests are being built). Make the default "build-osgi-manifests" property value false, so that devs don't have to wait around the extra 90 seconds on every build (unlike in the latest patch, where the default value is true). Include solr/ and modules/ in the build modifications (unlike the latest patch, which excludes them). Cause the "build-osgi-manifests" property to be set to true when packaging releases.
          Hide
          Steve Rowe added a comment -

          Is comment-13113588 still the only problem with getting this committed?

          No. comment-13113617 and comment-13124118 mention other problems.

          Show
          Steve Rowe added a comment - Is comment-13113588 still the only problem with getting this committed? No. comment-13113617 and comment-13124118 mention other problems.
          Hide
          Ryan Hill added a comment -

          Is comment-13113588 still the only problem with getting this committed?

          Show
          Ryan Hill added a comment - Is comment-13113588 still the only problem with getting this committed?
          Hide
          Steve Rowe added a comment - - edited

          Luca, you chose to implement your changes in a form that I will not commit - I think I was pretty clear about that. You too are free to change it. Leaving it to me means that it will be a while before it gets committed, since I have 10 other things I care about more. Cheers

          Show
          Steve Rowe added a comment - - edited Luca, you chose to implement your changes in a form that I will not commit - I think I was pretty clear about that. You too are free to change it. Leaving it to me means that it will be a while before it gets committed, since I have 10 other things I care about more. Cheers
          Hide
          Luca Stancapiano added a comment -

          I tried it and it works and I like it. You are free to change it. Cheers

          Show
          Luca Stancapiano added a comment - I tried it and it works and I like it. You are free to change it. Cheers
          Hide
          Steve Rowe added a comment -

          Luca, you dropped the changes from my patch to solr/ and modules/. Please put them back.

          I add a new property 'development'. This property could be used switch many other cases that can decrease the performances during the build.

          As I have stated previously, I don't like this idea.

          The default build with no properties specified should be development mode (i.e., don't do the extra work needed to build OSGi manifests). The Lucene/Solr build is for the developers; it must be as fast as possible by default.

          There should be a property named "build.osgi.manifests" or something similar that says what's happening, rather than hiding behind some anonymous "publish mode". That is, don't call the property "development" or "publish.mode". OSGi manifest building will be the only optional performance-decreasing element in the build, so there is no reason to generalize it at this point.

          Show
          Steve Rowe added a comment - Luca, you dropped the changes from my patch to solr/ and modules/ . Please put them back. I add a new property 'development'. This property could be used switch many other cases that can decrease the performances during the build. As I have stated previously, I don't like this idea. The default build with no properties specified should be development mode (i.e., don't do the extra work needed to build OSGi manifests). The Lucene/Solr build is for the developers; it must be as fast as possible by default. There should be a property named "build.osgi.manifests" or something similar that says what's happening, rather than hiding behind some anonymous "publish mode". That is, don't call the property "development" or "publish.mode". OSGi manifest building will be the only optional performance-decreasing element in the build, so there is no reason to generalize it at this point.
          Hide
          Luca Stancapiano added a comment - - edited

          I submit the new patch including the switch for the osgi compiler. I add a new property 'development'. This property could be used switch many other cases that can decrease the performances during the build. Here an example:

          ant clean default
          

          It will start in 'publish' mode. Actually only the osgi compile is added.

          ant clean default -Ddevelopment=true
          

          It will start in 'development' mode. The osgi compile will be removed and used the default manifest.mf

          ant clean default -Ddevelopment=
          

          Same thing as before

          Show
          Luca Stancapiano added a comment - - edited I submit the new patch including the switch for the osgi compiler. I add a new property 'development'. This property could be used switch many other cases that can decrease the performances during the build. Here an example: ant clean default It will start in 'publish' mode. Actually only the osgi compile is added. ant clean default -Ddevelopment= true It will start in 'development' mode. The osgi compile will be removed and used the default manifest.mf ant clean default -Ddevelopment= Same thing as before
          Hide
          Luca Stancapiano added a comment -

          The risk for a property for a single functionality is that someone can forget it during the publishing. Surely I will add it but I think a global property (like 'release' or 'dev') that starts sub-tasks (osgi bundle in this case) too can be useful

          Show
          Luca Stancapiano added a comment - The risk for a property for a single functionality is that someone can forget it during the publishing. Surely I will add it but I think a global property (like 'release' or 'dev') that starts sub-tasks (osgi bundle in this case) too can be useful
          Hide
          Steve Rowe added a comment -

          I imagine a property like 'dev' that excludes all heavy builds if it is not just ready. So it could be used for future ant tasks

          I disagree. The name of the property should tell people what it's for. 'dev' is the opposite of that. Also, functionality like this should be individually configurable.

          Show
          Steve Rowe added a comment - I imagine a property like 'dev' that excludes all heavy builds if it is not just ready. So it could be used for future ant tasks I disagree. The name of the property should tell people what it's for. 'dev' is the opposite of that. Also, functionality like this should be individually configurable.
          Hide
          Luca Stancapiano added a comment -

          I imagine a property like 'dev' that excludes all heavy builds if it is not just ready. So it could be used for future ant tasks

          Show
          Luca Stancapiano added a comment - I imagine a property like 'dev' that excludes all heavy builds if it is not just ready. So it could be used for future ant tasks
          Hide
          Steve Rowe added a comment -

          Because OSGi manifest creation is so slow, I will not commit this patch as-is.

          I think a good compromise would be to rewrite the make-manifest target to only make OSGi-conformant manifests either when a system property has been set from the command line or when generating release artifacts. Or maybe only when generating release artifacts?

          Show
          Steve Rowe added a comment - Because OSGi manifest creation is so slow, I will not commit this patch as-is. I think a good compromise would be to rewrite the make-manifest target to only make OSGi-conformant manifests either when a system property has been set from the command line or when generating release artifacts. Or maybe only when generating release artifacts?
          Hide
          Luca Stancapiano added a comment -

          it takes time because the bndlib is tied to the compilation of the sources. bndlib navigates recursively in the classpath of each module. More the dependencies are lot, more time is used. It is important to create the list of all the dependent classes

          Show
          Luca Stancapiano added a comment - it takes time because the bndlib is tied to the compilation of the sources. bndlib navigates recursively in the classpath of each module. More the dependencies are lot, more time is used. It is important to create the list of all the dependent classes
          Hide
          Robert Muir added a comment -

          Steven and I got to the bottom of this: the issue was the ant license files in lucene/lib. These have a ® sign encoded in ISO-8859-1.

          Show
          Robert Muir added a comment - Steven and I got to the bottom of this: the issue was the ant license files in lucene/lib. These have a ® sign encoded in ISO-8859-1.
          Hide
          Steve Rowe added a comment -

          Ouch: any idea why it takes so long? I think adding 153 seconds to the compilation time is very painful for developers.

          AFAICT, the bnd tool visits everything in the classpath and records everything it finds in the manifest. Luca, maybe you can add some more info here?

          FYI: I had encoding problems with the patch (its ISO-8859-1 not UTF-8?)

          How did you experience these problems? I generated it using (Cygwin's) svn diff on Windows 7. I skimmed the patch and didn't see any above-ASCII chars. The patch applies without complaint using Cygwin's patch.

          Show
          Steve Rowe added a comment - Ouch: any idea why it takes so long? I think adding 153 seconds to the compilation time is very painful for developers. AFAICT, the bnd tool visits everything in the classpath and records everything it finds in the manifest. Luca, maybe you can add some more info here? FYI: I had encoding problems with the patch (its ISO-8859-1 not UTF-8?) How did you experience these problems? I generated it using (Cygwin's) svn diff on Windows 7. I skimmed the patch and didn't see any above-ASCII chars. The patch applies without complaint using Cygwin's patch.
          Hide
          Robert Muir added a comment -

          I measured OSGI-compliant manifest creation time for all 38 non-example jars (and the Solr war), and the total cost was 153.5 seconds, which is about 4 seconds per bundle on average.

          Ouch: any idea why it takes so long? I think adding 153 seconds to the compilation time is very painful for developers.

          FYI: I had encoding problems with the patch (its ISO-8859-1 not UTF-8?)

          Show
          Robert Muir added a comment - I measured OSGI-compliant manifest creation time for all 38 non-example jars (and the Solr war), and the total cost was 153.5 seconds, which is about 4 seconds per bundle on average. Ouch: any idea why it takes so long? I think adding 153 seconds to the compilation time is very painful for developers. FYI: I had encoding problems with the patch (its ISO-8859-1 not UTF-8?)
          Hide
          Steve Rowe added a comment -

          Here's the correct patch.

          Show
          Steve Rowe added a comment - Here's the correct patch.
          Hide
          Steve Rowe added a comment -

          This patch fixes various problems with the created manifests.

          I measured OSGI-compliant manifest creation time for all 38 non-example jars (and the Solr war), and the total cost was 153.5 seconds, which is about 4 seconds per bundle on average.

          I also measured manifest creation time without the OSGI stuff, and it's basically instantaneous: single-digit milliseconds on average per jar/war.

          Maybe 4 seconds per jar/war is okay? I'll wait a day or two before I commit so people can object or suggest alternatives.

          One alternative to always building the OSGI manifests would be to only build them when preparing a release.

          Show
          Steve Rowe added a comment - This patch fixes various problems with the created manifests. I measured OSGI-compliant manifest creation time for all 38 non-example jars (and the Solr war), and the total cost was 153.5 seconds, which is about 4 seconds per bundle on average. I also measured manifest creation time without the OSGI stuff, and it's basically instantaneous: single-digit milliseconds on average per jar/war. Maybe 4 seconds per jar/war is okay? I'll wait a day or two before I commit so people can object or suggest alternatives. One alternative to always building the OSGI manifests would be to only build them when preparing a release.
          Hide
          Luca Stancapiano added a comment -

          Hi Stewen

          I've done the timing adding the two tstamp properties, so if you start ant will see in the log something as:

          Time starting for the build of manifest.mf: 10:55.50.186
          .....
          Time ending for the build of manifest.mf: 10:55.51.821

          The time for each module changes from 1 to a mx of 5 seconds according the number of dependencies during the compilation.

          About the second point you must not care about the wars because war and jar are read in the same manner in a OSGI repository. External utilities can add jobs to recognize Servlet, EJB or other javaee components but this work is delegated to the OSGI repository.

          I've told about javaee containers and I'm sure you know them.
          A package (jar or war) containing the OSGI informations is called 'bundle'.

          So, don't care about the war. They will be deployed without problems in a OSGI repository!!

          Show
          Luca Stancapiano added a comment - Hi Stewen I've done the timing adding the two tstamp properties, so if you start ant will see in the log something as: Time starting for the build of manifest.mf: 10:55.50.186 ..... Time ending for the build of manifest.mf: 10:55.51.821 The time for each module changes from 1 to a mx of 5 seconds according the number of dependencies during the compilation. About the second point you must not care about the wars because war and jar are read in the same manner in a OSGI repository. External utilities can add jobs to recognize Servlet, EJB or other javaee components but this work is delegated to the OSGI repository. I've told about javaee containers and I'm sure you know them. A package (jar or war) containing the OSGI informations is called 'bundle'. So, don't care about the war. They will be deployed without problems in a OSGI repository!!
          Hide
          Steve Rowe added a comment -

          Hi Luca,

          I'll take a look at your new patch today or tomorrow.

          Have you done any timings yet?

          I don't understand a couple of things you wrote:

          War files in OSGI are worked as the jar files.

          Do you mean that OSGI treats .war files the same as .jar files?

          If the OSGI repository has functionalities to work with containers, it takes the informations directly by the bundle. The MANIFEST.MF file doesn't include informations about containers.

          What is a container? What is a bundle? Why does it matter that MANIFEST.MF does not include information about containers? How are these things related to the other topics under discussion on this issue? (I wasn't kidding when I wrote that I know nothing about OSGi.)

          Show
          Steve Rowe added a comment - Hi Luca, I'll take a look at your new patch today or tomorrow. Have you done any timings yet? I don't understand a couple of things you wrote: War files in OSGI are worked as the jar files. Do you mean that OSGI treats .war files the same as .jar files? If the OSGI repository has functionalities to work with containers, it takes the informations directly by the bundle. The MANIFEST.MF file doesn't include informations about containers. What is a container? What is a bundle? Why does it matter that MANIFEST.MF does not include information about containers? How are these things related to the other topics under discussion on this issue? (I wasn't kidding when I wrote that I know nothing about OSGi.)
          Hide
          Luca Stancapiano added a comment - - edited

          Hi Steven,

          I send a new updated patch.

          I added two new stamp properties in the build-manifest macro (start.touch.time and end.touch.time) that log the milliseconds of the process.

          War files in OSGI are worked as the jar files. If the OSGI repository has functionalities to work with containers, it takes the informations directly by the bundle. The MANIFEST.MF file doesn't include informations about containers.

          I added the bnd library from http://dl.dropbox.com/u/2590603/bnd/biz.aQute.bndlib.jar (actually in the dropbox there is the only version for ant. See: http://www.aqute.biz/Bnd/Download) and added it to the ant classpath how for the 'generate-maven-artifacts' target.

          Here the responses to the tasks:

          1 - checked the box to grant the Apache license.

          2 - Renamed the patch according the convetion.

          3 - Deleted the bnd configuration for solr. Now only the build-manifest macro declared in the common-build.xml of lucene project is used. But I was forced to declare the attributes @

          {title}

          and @

          {implementation.title}

          as properties inside the build-manifest macro, else they didn't seen in the external file lucene.bnd.

          4 - I see the correct value of $

          {bnd.project.description}

          because the property is created through the configuration : <xmlproperty file="$

          {ant.file}

          " collapseAttributes="true" prefix="bnd"/> inside the build-manifest macro. Maybe I didn't added all in the previous patch. Let me know if the problem persists.

          5 - I excluded the DSTAMP, TSTAMP, and TODAY properties by the bnd configuration through the property: -removeheaders . The main problem is that the bnd ant task takes all the ant properties starting with an uppercased lecter and add them without ask. Should be a bnd property -inherit (true/false) that tells if import the ant properties but it doesn't work. This problem is signed in: https://github.com/bnd/bnd/issues/72. An other important thing is that the 'Name' ant property declared in some build.xml is not accepted by the bnd ant task. In the bnd ant task code there is an hard exception if the 'Name' property is found:

          			if (header.equalsIgnoreCase("Name")) {
          				error("Your bnd file contains a header called 'Name'. This interferes with the manifest name section.");
          				continue;
          			}
          

          So I was forced to rename the 'Name' property and its references in 'LuceneName'

          6 - Added the $

          {user.name}

          property in the Implementation-Version manifest property

          7 - Renamed the Bundle-DocUR property to Bundle-DocURL

          Show
          Luca Stancapiano added a comment - - edited Hi Steven, I send a new updated patch. I added two new stamp properties in the build-manifest macro (start.touch.time and end.touch.time) that log the milliseconds of the process. War files in OSGI are worked as the jar files. If the OSGI repository has functionalities to work with containers, it takes the informations directly by the bundle. The MANIFEST.MF file doesn't include informations about containers. I added the bnd library from http://dl.dropbox.com/u/2590603/bnd/biz.aQute.bndlib.jar (actually in the dropbox there is the only version for ant. See: http://www.aqute.biz/Bnd/Download ) and added it to the ant classpath how for the 'generate-maven-artifacts' target. Here the responses to the tasks: 1 - checked the box to grant the Apache license. 2 - Renamed the patch according the convetion. 3 - Deleted the bnd configuration for solr. Now only the build-manifest macro declared in the common-build.xml of lucene project is used. But I was forced to declare the attributes @ {title} and @ {implementation.title} as properties inside the build-manifest macro, else they didn't seen in the external file lucene.bnd. 4 - I see the correct value of $ {bnd.project.description} because the property is created through the configuration : <xmlproperty file="$ {ant.file} " collapseAttributes="true" prefix="bnd"/> inside the build-manifest macro. Maybe I didn't added all in the previous patch. Let me know if the problem persists. 5 - I excluded the DSTAMP, TSTAMP, and TODAY properties by the bnd configuration through the property: -removeheaders . The main problem is that the bnd ant task takes all the ant properties starting with an uppercased lecter and add them without ask. Should be a bnd property -inherit (true/false) that tells if import the ant properties but it doesn't work. This problem is signed in: https://github.com/bnd/bnd/issues/72 . An other important thing is that the 'Name' ant property declared in some build.xml is not accepted by the bnd ant task. In the bnd ant task code there is an hard exception if the 'Name' property is found: if (header.equalsIgnoreCase( "Name" )) { error( "Your bnd file contains a header called 'Name'. This interferes with the manifest name section." ); continue ; } So I was forced to rename the 'Name' property and its references in 'LuceneName' 6 - Added the $ {user.name} property in the Implementation-Version manifest property 7 - Renamed the Bundle-DocUR property to Bundle-DocURL
          Hide
          Steve Rowe added a comment -

          Hi Luca,

          I downloaded the bnd jar and applied your patch, then ran ant jar-core under lucene/. This produced a jar, and didn't seem too awfully slow. The generated MANIFEST.MF looks mostly okay - see below for some issues.

          Can you do some timings with and without the OSGI stuff for creating jars? If it doesn't take much extra time to produce the OSGi-compatible manifests, I think it would be okay to always produce them.

          Your patch doesn't have any special handling for Solr's war file manifest - does OSGi care about war files? (I know nothing about OSGi.)

          The bnd jar is Apache licensed, so we can put it into our repository, rather than require people to download the jar. You can see an example of this in lucene/build.xml and lucene/common-build.xml with the Maven Ant Tasks jar - take a look at the definitions of the generate-maven-artifacts target and the maven-ant-tasks.classpath path.

          Some stuff that needs to be addressed before this can be committed:

          1. You didn't check the box to grant the Apache license to your most recent patch; we can't commit your work unless you do this.
          2. Please follow the patch naming convention when you name patches; e.g. for this issue, the patch should be named LUCENE-3167.patch. See http://wiki.apache.org/lucene-java/HowToContribute#Creating_a_patch for more info.
          3. Solr's manifests are built twice - the jarify macro invokes the build-manifest macro. The jarify macro may need more parameters to be able to handle both Solr and Lucene needs.
          4. In the generated MANIFEST.MF, the Bundle-Description value is not interpolated properly - here's what I get: Bundle-Description: ${bnd.project.description}
          5. In the generated MANIFEST.MF, DSTAMP, TSTAMP, and TODAY each have entries, but they don't appear in the lucene.bnd template - if possible, these should be eliminated.
          6. In lucene.bnd, you have excluded ${user.name} from the Implementation-Version value - please re-sync with the unpatched value.
          7. In lucene.bnd, should Bundle-DocUR be Bundle-DocURL?
          Show
          Steve Rowe added a comment - Hi Luca, I downloaded the bnd jar and applied your patch, then ran ant jar-core under lucene/ . This produced a jar, and didn't seem too awfully slow. The generated MANIFEST.MF looks mostly okay - see below for some issues. Can you do some timings with and without the OSGI stuff for creating jars? If it doesn't take much extra time to produce the OSGi-compatible manifests, I think it would be okay to always produce them. Your patch doesn't have any special handling for Solr's war file manifest - does OSGi care about war files? (I know nothing about OSGi.) The bnd jar is Apache licensed, so we can put it into our repository, rather than require people to download the jar. You can see an example of this in lucene/build.xml and lucene/common-build.xml with the Maven Ant Tasks jar - take a look at the definitions of the generate-maven-artifacts target and the maven-ant-tasks.classpath path. Some stuff that needs to be addressed before this can be committed: You didn't check the box to grant the Apache license to your most recent patch; we can't commit your work unless you do this. Please follow the patch naming convention when you name patches; e.g. for this issue, the patch should be named LUCENE-3167 .patch . See http://wiki.apache.org/lucene-java/HowToContribute#Creating_a_patch for more info. Solr's manifests are built twice - the jarify macro invokes the build-manifest macro. The jarify macro may need more parameters to be able to handle both Solr and Lucene needs. In the generated MANIFEST.MF , the Bundle-Description value is not interpolated properly - here's what I get: Bundle-Description: ${bnd.project.description } In the generated MANIFEST.MF , DSTAMP , TSTAMP , and TODAY each have entries, but they don't appear in the lucene.bnd template - if possible, these should be eliminated. In lucene.bnd , you have excluded ${user.name } from the Implementation-Version value - please re-sync with the unpatched value. In lucene.bnd , should Bundle-DocUR be Bundle-DocURL ?
          Hide
          Luca Stancapiano added a comment -

          I send a new patch.

          • reorganized with the new update of the common-build.xml in lucene and solr
          • Configurated bnd task so we don't need more to create a jar where extract the new MANIFEST.MF
          Show
          Luca Stancapiano added a comment - I send a new patch. reorganized with the new update of the common-build.xml in lucene and solr Configurated bnd task so we don't need more to create a jar where extract the new MANIFEST.MF
          Hide
          Luca Stancapiano added a comment -

          I created an issue on https://github.com/bnd/bnd/issues/70 to parametrize the bnd ant task

          Show
          Luca Stancapiano added a comment - I created an issue on https://github.com/bnd/bnd/issues/70 to parametrize the bnd ant task
          Hide
          Luca Stancapiano added a comment -

          I expose this patch:

          <macrodef name="build-manifest" description="Builds a manifest file">
          <attribute name="title" default="Lucene Search Engine: $

          {ant.project.name}" />
          <attribute name="bndtempDir" default="${build.dir}/temp"/>
          <sequential>
          <xmlproperty file="${ant.file}" collapseAttributes="true" prefix="bnd"/>
          <property name="bndclasspath" refid="classpath"/>
          <taskdef resource="aQute/bnd/ant/taskdef.properties" />
          <mkdir dir="@{bndtempDir}"/>
          <bnd
          classpath="${bndclasspath}"
          eclipse="false"
          failok="false"
          exceptions="true"
          files="${common.dir}/lucene.bnd"
          output="@{bndtempDir}/${final.name}-temp.jar" />
          <copy todir="${common.dir}/build" flatten="true">
          <resources>
          <url url="jar:file://@{bndtempDir}/${final.name}-temp.jar!/META-INF/MANIFEST.MF"/>
          </resources>
          </copy>
          </sequential>
          </macrodef>

          It rewrites the build-manifest macrodef because bndlib cannot append the information of the manifest.mf. I moved the information of the manifest in the lucene.bnd file appending the new osgi info:


          Export-Package: *;-split-package:=merge-first
          Specification-Title: Lucene Search Engine: ${ant.project.name}

          Specification-Version: $

          {spec.version}

          Specification-Vendor: The Apache Software Foundation
          Implementation-Title: org.apache.lucene
          Implementation-Version: $

          {version} ${svnversion} - ${DSTAMP} ${TSTAMP}
          Implementation-Vendor: The Apache Software Foundation
          X-Compile-Source-JDK: ${javac.source}
          X-Compile-Target-JDK: ${javac.target}
          Bundle-License: http://www.apache.org/licenses/LICENSE-2.0.txt
          Bundle-SymbolicName: org.apache.lucene.${name}
          Bundle-Name: Lucene Search Engine: ${ant.project.name}
          Bundle-Vendor: The Apache Software Foundation
          Bundle-Version: ${version}

          Bundle-Description: $

          {bnd.project.description}

          Bundle-DocUR: http://www.apache.org/

          I tested on lucene and solr modules and all jars are created with the correct OSGI info in the manifest.mf. Unluckily bndlib is not flexible so if you use bndlib you are forced to:

          • precompile the classes
          • create a temp directory with a temporary jar
          • extract the new manifest from the jar and put it in the shared directory
          Show
          Luca Stancapiano added a comment - I expose this patch: <macrodef name="build-manifest" description="Builds a manifest file"> <attribute name="title" default="Lucene Search Engine: $ {ant.project.name}" /> <attribute name="bndtempDir" default="${build.dir}/temp"/> <sequential> <xmlproperty file="${ant.file}" collapseAttributes="true" prefix="bnd"/> <property name="bndclasspath" refid="classpath"/> <taskdef resource="aQute/bnd/ant/taskdef.properties" /> <mkdir dir="@{bndtempDir}"/> <bnd classpath="${bndclasspath}" eclipse="false" failok="false" exceptions="true" files="${common.dir}/lucene.bnd" output="@{bndtempDir}/${final.name}-temp.jar" /> <copy todir="${common.dir}/build" flatten="true"> <resources> <url url="jar: file://@{bndtempDir}/$ {final.name}-temp.jar!/META-INF/MANIFEST.MF"/> </resources> </copy> </sequential> </macrodef> It rewrites the build-manifest macrodef because bndlib cannot append the information of the manifest.mf. I moved the information of the manifest in the lucene.bnd file appending the new osgi info: Export-Package: *;-split-package:=merge-first Specification-Title: Lucene Search Engine: ${ant.project.name} Specification-Version: $ {spec.version} Specification-Vendor: The Apache Software Foundation Implementation-Title: org.apache.lucene Implementation-Version: $ {version} ${svnversion} - ${DSTAMP} ${TSTAMP} Implementation-Vendor: The Apache Software Foundation X-Compile-Source-JDK: ${javac.source} X-Compile-Target-JDK: ${javac.target} Bundle-License: http://www.apache.org/licenses/LICENSE-2.0.txt Bundle-SymbolicName: org.apache.lucene.${name} Bundle-Name: Lucene Search Engine: ${ant.project.name} Bundle-Vendor: The Apache Software Foundation Bundle-Version: ${version} Bundle-Description: $ {bnd.project.description} Bundle-DocUR: http://www.apache.org/ I tested on lucene and solr modules and all jars are created with the correct OSGI info in the manifest.mf. Unluckily bndlib is not flexible so if you use bndlib you are forced to: precompile the classes create a temp directory with a temporary jar extract the new manifest from the jar and put it in the shared directory
          Hide
          Luca Stancapiano added a comment -

          It depends by the structure of your templates. Can you tell me how is organized the tree of the template files?

          Show
          Luca Stancapiano added a comment - It depends by the structure of your templates. Can you tell me how is organized the tree of the template files?
          Hide
          Gunnar Wagenknecht added a comment -

          The approach I implemented in my patch (attached to LUCENE-1344) used template files for BND. I wonder if both - Ant as well as Maven could use those files.

          Show
          Gunnar Wagenknecht added a comment - The approach I implemented in my patch (attached to LUCENE-1344 ) used template files for BND. I wonder if both - Ant as well as Maven could use those files.
          Hide
          Luca Stancapiano added a comment -

          Some OSGI informations are inside the pom.xml . So if we want an automatism to create the OSGI attributes in the MANIFEST.MF, we have to read inside the pom.xml.template files.

          Theese are the OSGI informations to add in the MANIFEST:

          Bundle-License: http://www.apache.org/licenses/LICENSE-2.0.txt (project.licenses.license.url in the parent pom.xml.template)
          Bundle-SymbolicName: org.apache.lucene.misc (project.groupId+project.artifactId in the pom.xml.template)
          Bundle-Name: Lucene Miscellaneous (project.name attribute in the pom.xml.template)
          Bundle-Vendor: The Apache Software Foundation (from the parent pom.xml.template)
          Bundle-Version: 4.0-SNAPSHOT ($version variable from ant)
          Bundle-Description: Miscellaneous Lucene extensions (project.description from pom.xml.template)
          Bundle-DocURL: http://www.apache.org/ (project.documentation.url in the parent pom.xml.template)

          Else we should duplicate the informations. What is the better road?

          Show
          Luca Stancapiano added a comment - Some OSGI informations are inside the pom.xml . So if we want an automatism to create the OSGI attributes in the MANIFEST.MF, we have to read inside the pom.xml.template files. Theese are the OSGI informations to add in the MANIFEST: Bundle-License: http://www.apache.org/licenses/LICENSE-2.0.txt (project.licenses.license.url in the parent pom.xml.template) Bundle-SymbolicName: org.apache.lucene.misc (project.groupId+project.artifactId in the pom.xml.template) Bundle-Name: Lucene Miscellaneous (project.name attribute in the pom.xml.template) Bundle-Vendor: The Apache Software Foundation (from the parent pom.xml.template) Bundle-Version: 4.0-SNAPSHOT ($version variable from ant) Bundle-Description: Miscellaneous Lucene extensions (project.description from pom.xml.template) Bundle-DocURL: http://www.apache.org/ (project.documentation.url in the parent pom.xml.template) Else we should duplicate the informations. What is the better road?
          Hide
          Luca Stancapiano added a comment -

          Here a updated version using the correct classpath:

          <property name="bndclasspath" refid="classpath"/>
          <taskdef resource="aQute/bnd/ant/taskdef.properties" />
          <bnd
          classpath="$

          {bndclasspath}

          "
          eclipse="false"
          failok="false"
          exceptions="true"
          files="$

          {common.dir}

          /lucene.bnd" />

          The ant classpath is different by the maven classpath so there are differences in the resulting 'Export-Package' variable in the MANIFEST.MF but both are ok

          Show
          Luca Stancapiano added a comment - Here a updated version using the correct classpath: <property name="bndclasspath" refid="classpath"/> <taskdef resource="aQute/bnd/ant/taskdef.properties" /> <bnd classpath="$ {bndclasspath} " eclipse="false" failok="false" exceptions="true" files="$ {common.dir} /lucene.bnd" /> The ant classpath is different by the maven classpath so there are differences in the resulting 'Export-Package' variable in the MANIFEST.MF but both are ok
          Hide
          Luca Stancapiano added a comment -

          A starting point is here:

          Downloading the biz.aQute.bnd.jar package from http://dl.dropbox.com/u/2590603/bnd/biz.aQute.bnd.jar and put it in the libs home of ant.

          Then add this:

          <bnd
          classpath="src"
          eclipse="false"
          failok="false"
          exceptions="true"
          files="$

          {common.dir}

          /lucene.bnd" />

          in the lucene_trunk/lucene/common-build.xml file inside:

          <macrodef name="build-manifest" description="Builds a manifest file">
          <attribute name="title" default="Lucene Search Engine: $

          {ant.project.name}

          " />
          <sequential>
          .....

          and in the same directory create a lucene.bnd file containing:

          Export-Package: *;-split-package:=merge-first

          Need testing

          Show
          Luca Stancapiano added a comment - A starting point is here: Downloading the biz.aQute.bnd.jar package from http://dl.dropbox.com/u/2590603/bnd/biz.aQute.bnd.jar and put it in the libs home of ant. Then add this: <bnd classpath="src" eclipse="false" failok="false" exceptions="true" files="$ {common.dir} /lucene.bnd" /> in the lucene_trunk/lucene/common-build.xml file inside: <macrodef name="build-manifest" description="Builds a manifest file"> <attribute name="title" default="Lucene Search Engine: $ {ant.project.name} " /> <sequential> ..... and in the same directory create a lucene.bnd file containing: Export-Package: *;-split-package:=merge-first Need testing

            People

            • Assignee:
              Unassigned
              Reporter:
              Luca Stancapiano
            • Votes:
              12 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated:

                Development