Derby
  1. Derby
  2. DERBY-4120

derbyclient.jar is not a complete OSGi bundle

    Details

    • Urgency:
      Normal
    • Issue & fix info:
      High Value Fix, Known fix
    • Bug behavior facts:
      Embedded/Client difference

      Description

      The manifest in derbyclient.jar is incomplete for usage in OSGi. There are no packages exported. This appears to only be a problem in standalone equinox (outside of eclipse). Note that I did not try other OSGi containers.

      The fix is relatively simple. Adding the Export-Package manifest header should take care of the problem.

      1. pom.xml
        5 kB
        Charlie Mordant
      2. osgi.bnd
        0.2 kB
        Charlie Mordant
      3. derby OSGI issue.jpg
        49 kB
        Charlie Mordant
      4. addToManifest.diff
        0.8 kB
        Bryan Pendleton

        Issue Links

          Activity

          Hide
          Myrna van Lunteren added a comment -

          bulk change to close all issues resolved but not closed and not changed since June 1, 2014.

          Show
          Myrna van Lunteren added a comment - bulk change to close all issues resolved but not closed and not changed since June 1, 2014.
          Hide
          Charlie Mordant added a comment -

          Just in case someone have the same need as I, or if you change your mind as a temporary solution (or even if Jigsaw never see the light...), find attached the uber jar pom and bundle plugin osgi complementary file.
          Even if it herds cats...

          Regards,

          Show
          Charlie Mordant added a comment - Just in case someone have the same need as I, or if you change your mind as a temporary solution (or even if Jigsaw never see the light...), find attached the uber jar pom and bundle plugin osgi complementary file. Even if it herds cats... Regards,
          Hide
          Rick Hillegas added a comment -

          That's a useful analysis, Christian. I agree that the factoring of the packages into jar files is broken. I don't think we're going to refactor Derby in the 10.11 timeframe. We may be able to fix this when we refactor Derby to take advantage of the Jigsaw work being done in Java 9. But that won't provide any near term relief for OSGi users. Thanks.

          Show
          Rick Hillegas added a comment - That's a useful analysis, Christian. I agree that the factoring of the packages into jar files is broken. I don't think we're going to refactor Derby in the 10.11 timeframe. We may be able to fix this when we refactor Derby to take advantage of the Jigsaw work being done in Java 9. But that won't provide any near term relief for OSGi users. Thanks.
          Hide
          Christian Schneider added a comment -

          If I understand correctly the issue is that derby and derby client have the package org.apache.derby.jdbc with different contents. This does not work in OSGi. So the first step should be to move the code to different packages. The client jar seems to contain less packages so it is probably easer to move these.

          If you want to stay compatible you could create a new package org.apache.derby.client.jdbc and copy the classes there while keeping them in the original place too with @Deprecated to mark that people should switch. Then you could only export the new package in the Manifest. So OSGi users can/have to use the new packages while other users have some time to switch until the next major version where you could remove the old package.

          Another option would be to add the client classes into the derby jar so people can simply either install derby or client and do not need both.

          What do you think?

          Show
          Christian Schneider added a comment - If I understand correctly the issue is that derby and derby client have the package org.apache.derby.jdbc with different contents. This does not work in OSGi. So the first step should be to move the code to different packages. The client jar seems to contain less packages so it is probably easer to move these. If you want to stay compatible you could create a new package org.apache.derby.client.jdbc and copy the classes there while keeping them in the original place too with @Deprecated to mark that people should switch. Then you could only export the new package in the Manifest. So OSGi users can/have to use the new packages while other users have some time to switch until the next major version where you could remove the old package. Another option would be to add the client classes into the derby jar so people can simply either install derby or client and do not need both. What do you think?
          Hide
          Charlie Mordant added a comment -

          Hi Rick,

          Many other framework use this solution as an easier solution than changing all their architecture: for example ActiveMQ community choose the name 'activemq-osgi' to wrap the entire amq fwk without taking care of their internal design.
          If it is something that is not acceptable for Derby, I think that it will be very difficult to find an OSGI compliant, see a simple picture resuming the problem in attachement

          Show
          Charlie Mordant added a comment - Hi Rick, Many other framework use this solution as an easier solution than changing all their architecture: for example ActiveMQ community choose the name 'activemq-osgi' to wrap the entire amq fwk without taking care of their internal design. If it is something that is not acceptable for Derby, I think that it will be very difficult to find an OSGI compliant, see a simple picture resuming the problem in attachement
          Hide
          Rick Hillegas added a comment -

          Charlie, I don't mean to send you on a wild goose chase here. What we're looking for is a change to the Derby build scripts which will produce the standard jar files with the enough OSGi metadata in the manifests to fix this problem. Maybe that's not possible. Aggregating all of the jars into a single super jar ball won't be a solution which we can use in a Derby release. Thanks.

          Show
          Rick Hillegas added a comment - Charlie, I don't mean to send you on a wild goose chase here. What we're looking for is a change to the Derby build scripts which will produce the standard jar files with the enough OSGi metadata in the manifests to fix this problem. Maybe that's not possible. Aggregating all of the jars into a single super jar ball won't be a solution which we can use in a Derby release. Thanks.
          Hide
          Charlie Mordant added a comment -

          Hi,

          I made a superbundle made of the aggregation of derby, derbyclient, derbynetwork and derbytools, exporting all their contents (let's call it derby-all). I tested it, and it works for my use-case:
          Starting a server, creating a xadatasource, tables and querying it.

          I'll publish it as soon as possible, then attach it with this ticket.
          It is one of the solutions for that kind of issues (and easy as possible: consists on just a pom.xml and another two lined file).

          In order to integrate well and easily in an OSGI environment Derby team/a good soul should have to rename the most part of derby packages: for example, suffixing all packages by .network in the derbynet jar, .client in the client one...
          One of the goals to be OSGI compliant is to avoid having the same package exported by two different jars (bundle).
          Once that stuff is done, you just have to declare the maven-bundle-plugin in your pom to have an out-of-box OSGI compliant manifest (but this kind of changes should preferably be for the 11 release )

          See you (very) soon!

          Show
          Charlie Mordant added a comment - Hi, I made a superbundle made of the aggregation of derby, derbyclient, derbynetwork and derbytools, exporting all their contents (let's call it derby-all). I tested it, and it works for my use-case: Starting a server, creating a xadatasource, tables and querying it. I'll publish it as soon as possible, then attach it with this ticket. It is one of the solutions for that kind of issues (and easy as possible: consists on just a pom.xml and another two lined file). In order to integrate well and easily in an OSGI environment Derby team/a good soul should have to rename the most part of derby packages: for example, suffixing all packages by .network in the derbynet jar, .client in the client one... One of the goals to be OSGI compliant is to avoid having the same package exported by two different jars (bundle). Once that stuff is done, you just have to declare the maven-bundle-plugin in your pom to have an out-of-box OSGI compliant manifest (but this kind of changes should preferably be for the 11 release ) See you (very) soon!
          Hide
          Rick Hillegas added a comment -

          None of the Derby committers are experts on OSGi. If someone can supply a patch and verify that it works in OSGi environments, then one of the committers can verify that it doesn't break our regression tests and check it in. If this is done in the next two weeks, then the patch will make its way into the upcoming 10.11 release. Thanks.

          Show
          Rick Hillegas added a comment - None of the Derby committers are experts on OSGi. If someone can supply a patch and verify that it works in OSGi environments, then one of the committers can verify that it doesn't break our regression tests and check it in. If this is done in the next two weeks, then the patch will make its way into the upcoming 10.11 release. Thanks.
          Hide
          Christian Schneider added a comment -

          I think a small tweak can solve this problem. Simply export and import the package org.apache.derby.jdbc in both jars. Then OSGi will make sure the package from one of the jars is exported and the exported one is then also used for the derbyclient and derby bundles internally. This avoids the classloading problems that can occur if you export a package from two jars.

          Show
          Christian Schneider added a comment - I think a small tweak can solve this problem. Simply export and import the package org.apache.derby.jdbc in both jars. Then OSGi will make sure the package from one of the jars is exported and the exported one is then also used for the derbyclient and derby bundles internally. This avoids the classloading problems that can occur if you export a package from two jars.
          Hide
          Charlie Mordant added a comment -

          Why is it resolved?

          It's not resolved at all! derby.jar and derbyclient.jar are both exporting org.apache.derby.jdbc package, thus make it impossible to have a derby db and a client on the same OSGI runtime...

          Show
          Charlie Mordant added a comment - Why is it resolved? It's not resolved at all! derby.jar and derbyclient.jar are both exporting org.apache.derby.jdbc package, thus make it impossible to have a derby db and a client on the same OSGI runtime...
          Hide
          Cristiano Gavião added a comment -

          The same problem is in version 10.9.1.0 of derbyclient. Not all packages are being exported.

          Show
          Cristiano Gavião added a comment - The same problem is in version 10.9.1.0 of derbyclient. Not all packages are being exported.
          Hide
          Christian Schneider added a comment -

          A typical use case where you want both jars is if you want to host a networked derby server and one or more derby clients in the same osgi container. I think this should be possible.

          Show
          Christian Schneider added a comment - A typical use case where you want both jars is if you want to host a networked derby server and one or more derby clients in the same osgi container. I think this should be possible.
          Hide
          Bryan Pendleton added a comment -

          Why do you want to use both derbyclient.jar and derby.jar at the same time?

          Show
          Bryan Pendleton added a comment - Why do you want to use both derbyclient.jar and derby.jar at the same time?
          Hide
          Simon Chemouil added a comment - - edited

          Using 10.6.1, we experienced a problem with the manifest of derbyclient.jar which exports org.apache.derby.jdbc, a package that is already exported by derby.jar. It is thus impossible to use them both at the same time, in the same OSGi container, because the resolver will do the package wiring with only one of these. In our case it prevented us from using the Embedded JDBC driver from derby.jar.

          Three fixes are possible:

          • change the name of the jdbc driver package of derbyclient (e.g org.apache.derby.client.jdbc). That's the cleanest solution, even though it might break code. The old package name could be kept for backwards compatibility but not exported (in the Export-Package manifest header), so only OSGi installations would be affected (read: it would be 100% backwards compatible for others, but it might be interesting to deprecate the classes in the old package).
          • make derbyclient a "fragment" bundle of derby.jar, so they would be merged at runtime. I don't think this is proper, because it would imply a dependency of derbyclient on derby (that doesn't currently exist), and I'm not sure it makes sense. Then again, I don't know derby at all . I'm just integrating code that's using it.
          • use the split-package directive to tell the OSGi framework to do more complex package wiring, this is not recommanded though, but it's a solution. See http://code9.com/2008/08/22/tip-split-packages-and-visibility/ (google has more on the topic)

          So I believe this bug should be re-opened. Thanks

          Show
          Simon Chemouil added a comment - - edited Using 10.6.1, we experienced a problem with the manifest of derbyclient.jar which exports org.apache.derby.jdbc, a package that is already exported by derby.jar. It is thus impossible to use them both at the same time, in the same OSGi container, because the resolver will do the package wiring with only one of these. In our case it prevented us from using the Embedded JDBC driver from derby.jar. Three fixes are possible: change the name of the jdbc driver package of derbyclient (e.g org.apache.derby.client.jdbc). That's the cleanest solution, even though it might break code. The old package name could be kept for backwards compatibility but not exported (in the Export-Package manifest header), so only OSGi installations would be affected (read: it would be 100% backwards compatible for others, but it might be interesting to deprecate the classes in the old package). make derbyclient a "fragment" bundle of derby.jar, so they would be merged at runtime. I don't think this is proper, because it would imply a dependency of derbyclient on derby (that doesn't currently exist), and I'm not sure it makes sense. Then again, I don't know derby at all . I'm just integrating code that's using it. use the split-package directive to tell the OSGi framework to do more complex package wiring, this is not recommanded though, but it's a solution. See http://code9.com/2008/08/22/tip-split-packages-and-visibility/ (google has more on the topic) So I believe this bug should be re-opened. Thanks
          Hide
          Myrna van Lunteren added a comment -

          Committed backport to 10.5 with revision 963799.
          Merge command was: svn merge -c 830545 https://svn.apache.org/repos/asf/db/derby/code/trunk

          I confirmed that the appropriate entry is present in the META-INF/MANIFEST.MF file in derbyclient.jar.

          Show
          Myrna van Lunteren added a comment - Committed backport to 10.5 with revision 963799. Merge command was: svn merge -c 830545 https://svn.apache.org/repos/asf/db/derby/code/trunk I confirmed that the appropriate entry is present in the META-INF/MANIFEST.MF file in derbyclient.jar.
          Hide
          Myrna van Lunteren added a comment -

          Reopening to backport to 10.5. Changing assignee from Bryan to myself for that.

          Show
          Myrna van Lunteren added a comment - Reopening to backport to 10.5. Changing assignee from Bryan to myself for that.
          Hide
          Bryan Pendleton added a comment -

          Committed to the trunk as revision 830545.

          I hope that this change is sufficient to enable derbyclient.jar to be used as an OSGi bundle.

          If not, please let us know what additional problems arise, and we can address those.

          Show
          Bryan Pendleton added a comment - Committed to the trunk as revision 830545. I hope that this change is sufficient to enable derbyclient.jar to be used as an OSGi bundle. If not, please let us know what additional problems arise, and we can address those.
          Hide
          Bryan Pendleton added a comment -

          Here's a simple diff to the master build.xml file which causes the Manifest.mf
          for derbyclient.jar to have the following additional lines:

          Bundle-SymbolicName: derbyclient
          DynamicImport-Package: *
          Export-Package: org.apache.derby.jdbc

          Regression tests with this modified derbyclient.jar file were clean.

          I see that we already had

          Bundle-Vendor: Apache Software Foundation
          Bundle-Name: Apache Derby 10.6
          Bundle-Version: 10.6.0000000.828905M

          Do we need to set

          Bundle-ManifestVersion:2

          I'm not sure what Bundle-ManifestVersion does, but I see it referenced here:
          http://publib.boulder.ibm.com/infocenter/forms/v3r0m0/index.jsp?topic=/com.ibm.help.forms.doc/forms_services_platform/fsp_t_creating_manifest_file.html

          According to this, if we don't set Bundle-ManifestVersion to 2, we may be
          recognized as a legacy bundle:
          http://markmail.org/message/lnfjsdgpoh2qj3li

          I'm also still uncertain about which packages we ought to list in the
          Export-Package line. In my own experience, I don't ever need to
          refer to any other classes from derbyclient.jar other than the packages
          in org.apache.derby.jdbc. In fact, for my applications I only use
          org.apache.derby.jdbc.ClientDriver directly, so that would be sufficient for me.

          To get some other examples, I tried grepping around in the .java files in the
          java/testing subdirectory of the trunk, to see if any of our existing regression
          test programs access other packages from derbyclient. There were only a handful
          of hits, and from what I can tell these occur in code which would NOT be
          written by typical client applications.

          So I'm tempted to leave the Export-Package line very short for now, and
          only add to it if we come across cases where an application needs to access
          one of these classes.

          -bash-2.05b$ find . -name '*.java' -exec grep org.apache.derby.client {} \; -print
          import org.apache.derby.client.ClientXid;
          ./jdbcapi/XATransactionTest.java
          if (c instanceof org.apache.derby.client.am.Connection) {
          ((org.apache.derby.client.am.Connection) c).
          ./jdbcapi/CacheSessionDataTest.java
          // org.apache.derby.client.net.EbcdicCcsidManager
          ./jdbcapi/AuthenticationTest.java

          s.startsWith("org.apache.derby.client"))
          ./jdbcapi/ParameterMappingTest.java
          import org.apache.derby.client.am.ByteArrayCombinerStream;
          ./derbynet/ByteArrayCombinerStreamTest.java
          import org.apache.derby.client.am.SqlException;
          import org.apache.derby.client.am.ClientMessageId;
          ./derbynet/SqlExceptionTest.java
          Show
          Bryan Pendleton added a comment - Here's a simple diff to the master build.xml file which causes the Manifest.mf for derbyclient.jar to have the following additional lines: Bundle-SymbolicName: derbyclient DynamicImport-Package: * Export-Package: org.apache.derby.jdbc Regression tests with this modified derbyclient.jar file were clean. I see that we already had Bundle-Vendor: Apache Software Foundation Bundle-Name: Apache Derby 10.6 Bundle-Version: 10.6.0000000.828905M Do we need to set Bundle-ManifestVersion:2 I'm not sure what Bundle-ManifestVersion does, but I see it referenced here: http://publib.boulder.ibm.com/infocenter/forms/v3r0m0/index.jsp?topic=/com.ibm.help.forms.doc/forms_services_platform/fsp_t_creating_manifest_file.html According to this, if we don't set Bundle-ManifestVersion to 2, we may be recognized as a legacy bundle: http://markmail.org/message/lnfjsdgpoh2qj3li I'm also still uncertain about which packages we ought to list in the Export-Package line. In my own experience, I don't ever need to refer to any other classes from derbyclient.jar other than the packages in org.apache.derby.jdbc. In fact, for my applications I only use org.apache.derby.jdbc.ClientDriver directly, so that would be sufficient for me. To get some other examples, I tried grepping around in the .java files in the java/testing subdirectory of the trunk, to see if any of our existing regression test programs access other packages from derbyclient. There were only a handful of hits, and from what I can tell these occur in code which would NOT be written by typical client applications. So I'm tempted to leave the Export-Package line very short for now, and only add to it if we come across cases where an application needs to access one of these classes. -bash-2.05b$ find . -name '*.java' -exec grep org.apache.derby.client {} \; -print import org.apache.derby.client.ClientXid; ./jdbcapi/XATransactionTest.java if (c instanceof org.apache.derby.client.am.Connection) { ((org.apache.derby.client.am.Connection) c). ./jdbcapi/CacheSessionDataTest.java // org.apache.derby.client.net.EbcdicCcsidManager ./jdbcapi/AuthenticationTest.java s.startsWith("org.apache.derby.client")) ./jdbcapi/ParameterMappingTest.java import org.apache.derby.client.am.ByteArrayCombinerStream; ./derbynet/ByteArrayCombinerStreamTest.java import org.apache.derby.client.am.SqlException; import org.apache.derby.client.am.ClientMessageId; ./derbynet/SqlExceptionTest.java
          Hide
          JJ Snyder added a comment -

          All packages that contain classes that clients may want to cast to the derby implementation must be exported. For example, if you think that a client application wants to cast the Statement to org.apache.derby.client.am.PreparedStatement40 then the org.apache.derby.client.am package must be exported.

          Exporting all of the packages gives client applications the option to directly access the classes in those packages. So unless a package is intended to be private then it should probably be exported

          Show
          JJ Snyder added a comment - All packages that contain classes that clients may want to cast to the derby implementation must be exported. For example, if you think that a client application wants to cast the Statement to org.apache.derby.client.am.PreparedStatement40 then the org.apache.derby.client.am package must be exported. Exporting all of the packages gives client applications the option to directly access the classes in those packages. So unless a package is intended to be private then it should probably be exported
          Hide
          Knut Anders Hatlen added a comment -

          I see that the list of exported packages in derby.jar is close to, but not exactly the same as the list of packages in the published API (http://db.apache.org/derby/javadoc/publishedapi/jdbc4/). Based on what you have found till now, do you think they should be identical?

          Exporting org.apache.derby.jdbc in derbyclient.jar sounds like a fine, focused fix to the problem reported here, so +1 from me. (But all I know about this is what I read from the links you provided, so I'm definitely not an authority.)

          Show
          Knut Anders Hatlen added a comment - I see that the list of exported packages in derby.jar is close to, but not exactly the same as the list of packages in the published API ( http://db.apache.org/derby/javadoc/publishedapi/jdbc4/ ). Based on what you have found till now, do you think they should be identical? Exporting org.apache.derby.jdbc in derbyclient.jar sounds like a fine, focused fix to the problem reported here, so +1 from me. (But all I know about this is what I read from the links you provided, so I'm definitely not an authority.)
          Hide
          Bryan Pendleton added a comment -

          After reading through the papers a bit, and looking at the actual contents of derbyclient.jar,
          it seems to me that the only thing we should put into the Manifest for derbyclient.jar is:

          Export-Package: org.apache.derby.jdbc

          I think that all the other packages in derbyclient.jar are internal, and the only externally-visible
          package should be the above package, which contains ClientDriver.class

          I'll try that change, as it is the smallest and most focused.

          Show
          Bryan Pendleton added a comment - After reading through the papers a bit, and looking at the actual contents of derbyclient.jar, it seems to me that the only thing we should put into the Manifest for derbyclient.jar is: Export-Package: org.apache.derby.jdbc I think that all the other packages in derbyclient.jar are internal, and the only externally-visible package should be the above package, which contains ClientDriver.class I'll try that change, as it is the smallest and most focused.
          Hide
          Bryan Pendleton added a comment -

          A little bit of research on the web:

          This article mentions both Export-Package and Import-Package:
          http://www.javaworld.com/javaworld/jw-03-2008/jw-03-osgi1.html?page=3

          This article also says that importing must be done correctly:
          http://www.osgi.org/blog/2007/04/importance-of-exporting-nd-importing.html

          This article presents the Export-Package information from the Eclipse POV:
          http://help.eclipse.org/help32/index.jsp?topic=/org.eclipse.platform.doc.isv/reference/misc/bundle_manifest.html

          It seems like the simplest change would be to put the identical Export-Package
          line into derbyclient.jar as is used for derby.jar.

          I don't understand what this would fix, but I can at least prototype this and
          verify that it doesn't break anything (that is revealed by the regression tests).

          Show
          Bryan Pendleton added a comment - A little bit of research on the web: This article mentions both Export-Package and Import-Package: http://www.javaworld.com/javaworld/jw-03-2008/jw-03-osgi1.html?page=3 This article also says that importing must be done correctly: http://www.osgi.org/blog/2007/04/importance-of-exporting-nd-importing.html This article presents the Export-Package information from the Eclipse POV: http://help.eclipse.org/help32/index.jsp?topic=/org.eclipse.platform.doc.isv/reference/misc/bundle_manifest.html It seems like the simplest change would be to put the identical Export-Package line into derbyclient.jar as is used for derby.jar. I don't understand what this would fix , but I can at least prototype this and verify that it doesn't break anything (that is revealed by the regression tests).
          Hide
          Bryan Pendleton added a comment -

          Where do I go to learn more about Export-Package and what it should contain?

          I did a bit of poking about in the 10.5.3.0 distribution of Derby, and I see:

          • derby.jar's Manifest.mf contains the following Export-Package information:

          Export-Package: org.apache.derby.authentication,org.apache.derby.datab
          ase,org.apache.derby.io,org.apache.derby.jdbc,org.apache.derby.vti

          • none of the other Derby jar files (derbyclient.jar, derbytools.jar, derbynet.jar, derbyrun.jar)
            seem to contain Export-Package lines

          So I guess my questions are:
          1) Should we add Export-Package to all the Derby jar files? Or just derbyclient.jar?
          2) What value should it contain? Is it the same for each Derby jar file? Or should it
          have a different value in derbyclient.jar than it does in derby.jar?

          Show
          Bryan Pendleton added a comment - Where do I go to learn more about Export-Package and what it should contain? I did a bit of poking about in the 10.5.3.0 distribution of Derby, and I see: derby.jar's Manifest.mf contains the following Export-Package information: Export-Package: org.apache.derby.authentication,org.apache.derby.datab ase,org.apache.derby.io,org.apache.derby.jdbc,org.apache.derby.vti none of the other Derby jar files (derbyclient.jar, derbytools.jar, derbynet.jar, derbyrun.jar) seem to contain Export-Package lines So I guess my questions are: 1) Should we add Export-Package to all the Derby jar files? Or just derbyclient.jar? 2) What value should it contain? Is it the same for each Derby jar file? Or should it have a different value in derbyclient.jar than it does in derby.jar?
          Hide
          Rick Hillegas added a comment -

          Triaged July 2, 2009: Marked as "Known Fix".

          Show
          Rick Hillegas added a comment - Triaged July 2, 2009: Marked as "Known Fix".

            People

            • Assignee:
              Bryan Pendleton
              Reporter:
              JJ Snyder
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development