Uploaded image for project: 'Felix'
  1. Felix
  2. FELIX-5592

Maven bundle plugin does not support Java 9 Multi-Release jars

    Details

      Description

      Log4j 2 currently packages its jars with the maven-bundle-plugin. To support Java 9 we are moving towards using Multi-Release jars to pick up the Java 9 features. However, when the Maven bundle plugin encouters classes in META-INF/versions/9 it emits an error message saying "Classes found in the wrong directory:", which is incorrect for a Multi-Release jar in Java 9.

        Issue Links

          Activity

          Hide
          karlpauls Karl Pauls added a comment -

          This should be fixed with the upcomming maven-bundle-plugin 3.4.0 release - where by "fixed" I mean it should at least allow to build with multi-release jars. Ultimately, we will need some spec level support to fully support multi-release jars (see the bnd issues mentioned) but that might be in the works.

          I'm resolving this issue as fixed for now as the problem should have gone away. When we know how spec level support might look like we need to open a new issue to implement the changes.

          Show
          karlpauls Karl Pauls added a comment - This should be fixed with the upcomming maven-bundle-plugin 3.4.0 release - where by "fixed" I mean it should at least allow to build with multi-release jars. Ultimately, we will need some spec level support to fully support multi-release jars (see the bnd issues mentioned) but that might be in the works. I'm resolving this issue as fixed for now as the problem should have gone away. When we know how spec level support might look like we need to open a new issue to implement the changes.
          Hide
          karlpauls Karl Pauls added a comment -

          Neil Bartlett, can you create an issue and do the update? If so, I can do a new release.

          Show
          karlpauls Karl Pauls added a comment - Neil Bartlett , can you create an issue and do the update? If so, I can do a new release.
          Hide
          nbartlett Neil Bartlett added a comment -

          Thanks Ralph. That last part is because the processing of module-info.class was fixed in bnd 3.4.0 but there has not been a release of maven-bundle-plugin. In the meantime bnd has released version 3.5.0.

          Can we have a new release of maven-bundle-plugin please?? Carsten Ziegeler

          Show
          nbartlett Neil Bartlett added a comment - Thanks Ralph. That last part is because the processing of module-info.class was fixed in bnd 3.4.0 but there has not been a release of maven-bundle-plugin. In the meantime bnd has released version 3.5.0. Can we have a new release of maven-bundle-plugin please?? Carsten Ziegeler
          Hide
          ralph.goers@dslextreme.com Ralph Goers added a comment -

          I also had to add

                  <dependencies>
                    <dependency>
                      <groupId>biz.aQute.bnd</groupId>
                      <artifactId>biz.aQute.bndlib</artifactId>
                      <version>3.5.0</version>
                    </dependency>
                  </dependencies>
          

          to my plugin declaration to ignore the maven-info.java file.

          Show
          ralph.goers@dslextreme.com Ralph Goers added a comment - I also had to add <dependencies> <dependency> <groupId>biz.aQute.bnd</groupId> <artifactId>biz.aQute.bndlib</artifactId> <version>3.5.0</version> </dependency> </dependencies> to my plugin declaration to ignore the maven-info.java file.
          Hide
          dmlloyd David M. Lloyd added a comment -

          Great tip Neil, thanks. That at least might be enough to let our builds actually complete.

          Show
          dmlloyd David M. Lloyd added a comment - Great tip Neil, thanks. That at least might be enough to let our builds actually complete.
          Hide
          nbartlett Neil Bartlett added a comment -

          You can continue using the maven-bundle-plugin with your multi-release JARs by adding the following configuration:

                    <configuration>
                      <instructions>
                        <_fixupmessages>"Classes found in the wrong directory";is:=warning</_fixupmessages>
                      </instructions>
                    </configuration>
          

          Note that the Java-9-specific parts will not be visible in OSGi, because OSGi will need enhancements to its runtime (the problem is that the Java 9 parts could have different dependencies, and OSGi does not support dependencies that are conditional upon the runtime Java version). Work on addressing this limitation is underway. In the meantime, the above snippet will allow you to keep using the maven-bundle-plugin to support OSGi users running on Java 8 and below.

          Show
          nbartlett Neil Bartlett added a comment - You can continue using the maven-bundle-plugin with your multi-release JARs by adding the following configuration: <configuration> <instructions> <_fixupmessages>"Classes found in the wrong directory";is:=warning</_fixupmessages> </instructions> </configuration> Note that the Java-9-specific parts will not be visible in OSGi, because OSGi will need enhancements to its runtime (the problem is that the Java 9 parts could have different dependencies, and OSGi does not support dependencies that are conditional upon the runtime Java version). Work on addressing this limitation is underway. In the meantime, the above snippet will allow you to keep using the maven-bundle-plugin to support OSGi users running on Java 8 and below.
          Hide
          nbartlett Neil Bartlett added a comment -

          The underlying issue to be fixed is in bnd, which is consumed by maven-bundle-plugin (and other tools). I have raised the following issue: https://github.com/bndtools/bnd/issues/2227

          Show
          nbartlett Neil Bartlett added a comment - The underlying issue to be fixed is in bnd, which is consumed by maven-bundle-plugin (and other tools). I have raised the following issue: https://github.com/bndtools/bnd/issues/2227
          Hide
          ralph.goers@dslextreme.com Ralph Goers added a comment -

          As of Log4j 2.9 Log4j has been using multi-release jars so this problem could be impacting users even if they aren't migrating to Java 9.

          Show
          ralph.goers@dslextreme.com Ralph Goers added a comment - As of Log4j 2.9 Log4j has been using multi-release jars so this problem could be impacting users even if they aren't migrating to Java 9.
          Hide
          dmlloyd David M. Lloyd added a comment -

          We (JBoss) are also moving to multi-release JARs for many of our projects. Those projects which use the maven-bundle-plugin will lose OSGi support until this issue is resolved.

          Show
          dmlloyd David M. Lloyd added a comment - We (JBoss) are also moving to multi-release JARs for many of our projects. Those projects which use the maven-bundle-plugin will lose OSGi support until this issue is resolved.
          Hide
          ralph.goers@dslextreme.com Ralph Goers added a comment -

          Interestingly, we want to use a multi-release jar for exactly the same reason as LWJGL. Log4j 2 has to walk the stack to get location information into the logs.

          A major problem with your proposal is we don't want more jars. We have an API jar and an impl jar and then jars for a whole bunch of optional things. We don't want users on Java 9 to have to include an extra jar to get better performance, nor do we want user's to have to pick and choose between jars that are compiled for their environment. Log4j 2 as a whole compiles with Java 7. To get the build to work I did have to create a maven module that compiles with Java 9, but the classes generated there are copied into either the API or impl jar as appropriate.

          This all works fine except for the Maven bundle plugin.

          Show
          ralph.goers@dslextreme.com Ralph Goers added a comment - Interestingly, we want to use a multi-release jar for exactly the same reason as LWJGL. Log4j 2 has to walk the stack to get location information into the logs. A major problem with your proposal is we don't want more jars. We have an API jar and an impl jar and then jars for a whole bunch of optional things. We don't want users on Java 9 to have to include an extra jar to get better performance, nor do we want user's to have to pick and choose between jars that are compiled for their environment. Log4j 2 as a whole compiles with Java 7. To get the build to work I did have to create a maven module that compiles with Java 9, but the classes generated there are copied into either the API or impl jar as appropriate. This all works fine except for the Maven bundle plugin.
          Hide
          io7m Mark Raynsford added a comment - - edited

          Er, to clarify, ServiceLoader isn't used in OSGi. Generally though, if a piece of code is written to work well with the ServiceLoader API, then it can be made to work under OSGi with next to no effort. With OSGi's "declarative services", it's just a couple of annotations that aren't retained in any form (they're used to generate a small piece of XML metadata that's injected into the jar file and used by the OSGi container).

          Show
          io7m Mark Raynsford added a comment - - edited Er, to clarify, ServiceLoader isn't used in OSGi. Generally though, if a piece of code is written to work well with the ServiceLoader API, then it can be made to work under OSGi with next to no effort. With OSGi's "declarative services", it's just a couple of annotations that aren't retained in any form (they're used to generate a small piece of XML metadata that's injected into the jar file and used by the OSGi container).
          Hide
          io7m Mark Raynsford added a comment -

          I don't want to be seen as strongly suggesting you should or shouldn't do anything: I don't know enough about your particular case. Some projects want to use multi-release jars to deliver code that is present on both (Java < 9) and (Java >= 9). Other projects want to use multi-release jars to deliver code that has extra functionality on (Java >= 9). An example of the latter is the LWJGL project: They use the new stack-walking API for performance reasons if it's available, but if it isn't, the code works but more slowly.

          If I was dealing with situations like the above, I think I'd do the following:

          1. Work out what kind of API my optional code needed. This is the "provider" API; the API that the main module uses to communicate with its own optional functionality. I'd define it as a set of interfaces and publish them in an API jar (call it "api.jar"). I'd write an OSGi manifest that exported the API, and a Java 9 module descriptor that exported the API in the same manner.

          2. Write a (Java < 9) implementation of that API and publish it in its own jar (call it "jdk8.jar"). I'd write an OSGi manifest that publishes my implementation as providing the services declared in "api.jar" and declare it as requiring a runtime environment of (Java < 9). I'd add META-INF/service/* files that publish the implementation as normal Java services.

          3. Write a (Java >= 9) implementation of that API and publish it in its own jar (call it "jdk9.jar"). I'd write an OSGi manifest that publishes my implementation as providing the services declared in "api.jar" and declare it as requiring a runtime environment of (Java >= 9). I'd add a Java 9 module descriptor that states that it provides the services defined in the API jar.

          I believe with this arrangement that an OSGi container would then automatically pick up the right implementation based on the current Java environment and the information in the manifest.

          Outside of an OSGi container, with a (Java < 9) environment and all of the jars placed on the class path, only the implementation in "jdk8.jar" will be picked up (because the "jdk9.jar" only declares services via the module descriptor).

          Outside of an OSGi container, with a (Java >= 9) environment and all of the jars placed on the module path, I believe only the "jdk9.jar" implementation will be picked up. I'm not exactly certain on this as there may be some interaction with Java 9's "automodules" system. Best case, only the (Java > 9) service implementation will be picked up. Worst case, they'll both be picked up and it'll be the responsibility of the service consumer to pick the "better" implementation. You'd need a method defined on the API to allow implementations to be ranked in some manner.

          It's certainly more work than multi-release jars but, in my opinion it's a more disciplined approach. It makes optional and possibly platform-specific dependencies explicit and uses a well-understood API (ServiceLoader) to work with the code instead of, for example, resorting to class path and reflection hacks. This is the standard way to represent optional functionality (or multiple implementations) in OSGi and I would guess that it'll become the recommended way to do things in Java 9 as it appears to be heavily influenced by the methodologies people have arrived at on the OSGi side. Java 9's module system appears to be a simpler and far less dynamic clone of OSGi in most aspects, so the same techniques tend to apply.

          Show
          io7m Mark Raynsford added a comment - I don't want to be seen as strongly suggesting you should or shouldn't do anything: I don't know enough about your particular case. Some projects want to use multi-release jars to deliver code that is present on both (Java < 9) and (Java >= 9). Other projects want to use multi-release jars to deliver code that has extra functionality on (Java >= 9). An example of the latter is the LWJGL project: They use the new stack-walking API for performance reasons if it's available, but if it isn't, the code works but more slowly. If I was dealing with situations like the above, I think I'd do the following: 1. Work out what kind of API my optional code needed. This is the "provider" API; the API that the main module uses to communicate with its own optional functionality. I'd define it as a set of interfaces and publish them in an API jar (call it "api.jar"). I'd write an OSGi manifest that exported the API, and a Java 9 module descriptor that exported the API in the same manner. 2. Write a (Java < 9) implementation of that API and publish it in its own jar (call it "jdk8.jar"). I'd write an OSGi manifest that publishes my implementation as providing the services declared in "api.jar" and declare it as requiring a runtime environment of (Java < 9). I'd add META-INF/service/* files that publish the implementation as normal Java services. 3. Write a (Java >= 9) implementation of that API and publish it in its own jar (call it "jdk9.jar"). I'd write an OSGi manifest that publishes my implementation as providing the services declared in "api.jar" and declare it as requiring a runtime environment of (Java >= 9). I'd add a Java 9 module descriptor that states that it provides the services defined in the API jar. I believe with this arrangement that an OSGi container would then automatically pick up the right implementation based on the current Java environment and the information in the manifest. Outside of an OSGi container, with a (Java < 9) environment and all of the jars placed on the class path, only the implementation in "jdk8.jar" will be picked up (because the "jdk9.jar" only declares services via the module descriptor). Outside of an OSGi container, with a (Java >= 9) environment and all of the jars placed on the module path, I believe only the "jdk9.jar" implementation will be picked up. I'm not exactly certain on this as there may be some interaction with Java 9's "automodules" system. Best case, only the (Java > 9) service implementation will be picked up. Worst case, they'll both be picked up and it'll be the responsibility of the service consumer to pick the "better" implementation. You'd need a method defined on the API to allow implementations to be ranked in some manner. It's certainly more work than multi-release jars but, in my opinion it's a more disciplined approach. It makes optional and possibly platform-specific dependencies explicit and uses a well-understood API (ServiceLoader) to work with the code instead of, for example, resorting to class path and reflection hacks. This is the standard way to represent optional functionality (or multiple implementations) in OSGi and I would guess that it'll become the recommended way to do things in Java 9 as it appears to be heavily influenced by the methodologies people have arrived at on the OSGi side. Java 9's module system appears to be a simpler and far less dynamic clone of OSGi in most aspects, so the same techniques tend to apply.
          Hide
          ralph.goers@dslextreme.com Ralph Goers added a comment -

          If I get what you are suggesting, the Java 9 replacement classes would need to have unique names or be in different packages. and then the code that wants to use these would have to use the ServiceLoader to locate the implementations and somehow pick the one for the current jdk. This seems like a lot more work than what is required of the application for multi-release support. On the upside, I suppose it would make debugging easier. But as far as I can tell, the only reason for doing it with the service loader is to make OSGi "happy".

          Show
          ralph.goers@dslextreme.com Ralph Goers added a comment - If I get what you are suggesting, the Java 9 replacement classes would need to have unique names or be in different packages. and then the code that wants to use these would have to use the ServiceLoader to locate the implementations and somehow pick the one for the current jdk. This seems like a lot more work than what is required of the application for multi-release support. On the upside, I suppose it would make debugging easier. But as far as I can tell, the only reason for doing it with the service loader is to make OSGi "happy".
          Hide
          io7m Mark Raynsford added a comment -

          I think the main issue is that OSGi doesn't have any way of representing "classes that are only loaded in a specific jvm environment". As I mentioned in that thread, the right way to do this is via services (both in OSGi and Java 9), but that's not much help for existing code that isn't structured as services.

          Show
          io7m Mark Raynsford added a comment - I think the main issue is that OSGi doesn't have any way of representing "classes that are only loaded in a specific jvm environment". As I mentioned in that thread, the right way to do this is via services (both in OSGi and Java 9), but that's not much help for existing code that isn't structured as services.
          Hide
          ralph.goers@dslextreme.com Ralph Goers added a comment - - edited

          Thanks, I just read that thread. I have no plans to subscribe to the bnd-tools mailing list so I can't comment there. Java 9 is scheduled to be released in a few months. As such Log4j 2 needs to have support in place for that. The spec for multi-release jars is not going to change and if it was going to that would have happened a year or more ago. It sounds like they are sticking their heads in the sand hoping the problem will go away. Instead it is going to get worse. To be honest, supporting Java 9 is a much higher priority to me than supporting OSGi.

          Show
          ralph.goers@dslextreme.com Ralph Goers added a comment - - edited Thanks, I just read that thread. I have no plans to subscribe to the bnd-tools mailing list so I can't comment there. Java 9 is scheduled to be released in a few months. As such Log4j 2 needs to have support in place for that. The spec for multi-release jars is not going to change and if it was going to that would have happened a year or more ago. It sounds like they are sticking their heads in the sand hoping the problem will go away. Instead it is going to get worse. To be honest, supporting Java 9 is a much higher priority to me than supporting OSGi.
          Hide
          io7m Mark Raynsford added a comment -

          It's the same root cause as this: https://issues.apache.org/jira/browse/FELIX-5527

          Show
          io7m Mark Raynsford added a comment - It's the same root cause as this: https://issues.apache.org/jira/browse/FELIX-5527

            People

            • Assignee:
              nbartlett Neil Bartlett
              Reporter:
              ralph.goers@dslextreme.com Ralph Goers
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development