Log4j 2
  1. Log4j 2
  2. LOG4J2-10

log4j 2.0 should work well with OSGi and Apache Felix

    Details

    • Type: Wish Wish
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 2.0-rc1
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      OSGi and specifically the Apache Felix implementation should be considered for framework services such as internal logging and configuration.

      log4j 2.0 should be able to be a provider of OSGi logging services.

      OSGi package visibility declarations should be used to distinguish between exported and explicitly supported APIs and implementation specific details.

      1. reduced-dependency-bundles.patch
        28 kB
        Timothy Ward
      2. reduced-dependency-bundles-2.patch
        33 kB
        Timothy Ward
      3. tests-and-version-ranges.patch
        36 kB
        Timothy Ward

        Activity

        Hide
        Curt Arnold added a comment -

        A few resources from browsing, haven't processed them:

        http://blog.kornr.net/index.php/2008/12/09/understanding-the-osgi-logging-service
        http://sling.apache.org/site/architecture.html (mention of OSGi Logging Service, log4j, slf4j and commons logging).
        http://wiki.ops4j.org/display/paxlogging/Pax+Logging

        Show
        Curt Arnold added a comment - A few resources from browsing, haven't processed them: http://blog.kornr.net/index.php/2008/12/09/understanding-the-osgi-logging-service http://sling.apache.org/site/architecture.html (mention of OSGi Logging Service, log4j, slf4j and commons logging). http://wiki.ops4j.org/display/paxlogging/Pax+Logging
        Hide
        Ralph Goers added a comment -

        Log4j 2 is using the maven-bundle-plugin to create osgi manifests. Whether these are correct needs to be validated.

        Show
        Ralph Goers added a comment - Log4j 2 is using the maven-bundle-plugin to create osgi manifests. Whether these are correct needs to be validated.
        Hide
        Pedro Lamarão added a comment -

        log4j 2 beta 2 includes an Import-Package for javax.jms; version="0.0.0" which won't be provided by out-of-the-box OSGi implementations.
        Trying to start the code bundle in a minimal Equinox target causes: org.osgi.framework.BundleException: The bundle "org.apache.commons.log4j-core_2.0.0.beta2 [6]" could not be resolved. Reason: Missing Constraint: Import-Package: javax.jms; version="0.0.0"

        Show
        Pedro Lamarão added a comment - log4j 2 beta 2 includes an Import-Package for javax.jms; version="0.0.0" which won't be provided by out-of-the-box OSGi implementations. Trying to start the code bundle in a minimal Equinox target causes: org.osgi.framework.BundleException: The bundle "org.apache.commons.log4j-core_2.0.0.beta2 [6] " could not be resolved. Reason: Missing Constraint: Import-Package: javax.jms; version="0.0.0"
        Hide
        Timothy Ward added a comment -

        I've looked at the log4j2 api and core bundles.

        The api bundle looks good. The packages are all exported (which is fine for an API bundle) and all of the exports have versions with sensible numbers. There are no imports, which will make the API bundle very easy to reuse.

        The core bundle has a few more issues. As Pedro Lamarão pointed out above, the log4j core jar adds lots of dependencies. There are several issues:

        1. The core bundle exports every single one of its packages. Are all of these classes public API? Any classes that aren't intended for public use should not be exported from the bundle.

        2. None of the package imports are versioned - this makes it very difficult to work out what else should be installed, and can cause significant compatibility problems

        3.The package imports are all mandatory, even though most users will not need any of the function. Things like lmax disruptor and javax jms are good examples of this. If I just want a standard RollingFileAppender then I shouldn't have to install things like Mongo or JPA.

        The first of these sets off lots of alarm bells, but as I'm not an expert in the log4j API (other than as a user of it!) I'm not really able to tell you any specific actions. It may be that all of those classes really are public API, in which case the existing packaging is right.

        The second of these is reasonably easy to fix as the maven bundle plugin scans any dependencies for OSGi version information and adds it automatically. Unfortunately the log4j2 build sets <excludeDependencies>true</excludeDependencies> in the maven bundle plugin configuration. This means that none of the dependency information is available when constructing the manifest. If you change this property and make sure that your dependencies are already OSGi bundles (for example, version 1.1.1 of the JMS 1.1. API from Geronimo is an OSGi bundle) then the maven bundle plugin will automatically fill in that information for you. You can then manually fix up anything that's left. One thing to note is that you may also need to tweak the export package/private package statements for your bundles to avoid pulling additional code into your bundle.

        The third of these indicates that the scope of the log4j core bundle is too big. It should be reasonably easy to break off bundles for the core.jpa, core.async, core.nosql.mongo, core.nosql.couch etc. By packaing these separately you can make it much easier to consume the log4j core bundle without dragging in the whole of the dependency tree. This is preferable to optional imports, as it prevents you from having to debug unpleasant NoClassDefFoundErrors at runtime.

        Show
        Timothy Ward added a comment - I've looked at the log4j2 api and core bundles. The api bundle looks good. The packages are all exported (which is fine for an API bundle) and all of the exports have versions with sensible numbers. There are no imports, which will make the API bundle very easy to reuse. The core bundle has a few more issues. As Pedro Lamarão pointed out above, the log4j core jar adds lots of dependencies. There are several issues: 1. The core bundle exports every single one of its packages. Are all of these classes public API? Any classes that aren't intended for public use should not be exported from the bundle. 2. None of the package imports are versioned - this makes it very difficult to work out what else should be installed, and can cause significant compatibility problems 3.The package imports are all mandatory, even though most users will not need any of the function. Things like lmax disruptor and javax jms are good examples of this. If I just want a standard RollingFileAppender then I shouldn't have to install things like Mongo or JPA. The first of these sets off lots of alarm bells, but as I'm not an expert in the log4j API (other than as a user of it!) I'm not really able to tell you any specific actions. It may be that all of those classes really are public API, in which case the existing packaging is right. The second of these is reasonably easy to fix as the maven bundle plugin scans any dependencies for OSGi version information and adds it automatically. Unfortunately the log4j2 build sets <excludeDependencies>true</excludeDependencies> in the maven bundle plugin configuration. This means that none of the dependency information is available when constructing the manifest. If you change this property and make sure that your dependencies are already OSGi bundles (for example, version 1.1.1 of the JMS 1.1. API from Geronimo is an OSGi bundle) then the maven bundle plugin will automatically fill in that information for you. You can then manually fix up anything that's left. One thing to note is that you may also need to tweak the export package/private package statements for your bundles to avoid pulling additional code into your bundle. The third of these indicates that the scope of the log4j core bundle is too big. It should be reasonably easy to break off bundles for the core.jpa, core.async, core.nosql.mongo, core.nosql.couch etc. By packaing these separately you can make it much easier to consume the log4j core bundle without dragging in the whole of the dependency tree. This is preferable to optional imports, as it prevents you from having to debug unpleasant NoClassDefFoundErrors at runtime.
        Hide
        Remko Popma added a comment -

        Thanks for the detailed help!
        Can you give an example of how to do #3?

        Show
        Remko Popma added a comment - Thanks for the detailed help! Can you give an example of how to do #3?
        Hide
        Timothy Ward added a comment -

        Remko Popma
        The simplest way I can think of to achieve #3 would be to split log4j-core into multiple bundles. A "main" bundle containing most of the log4j-core, and then some small extensions for each of the optional pieces. At a glance I would expect there to be an extension for "async", one for JPA, and two NoSQL (one for mongo and one for couch), but there may be others that could be easily extracted to reduce the number of package imports needed. I did wonder whether the jackson dependencies could be easily factored out, but it's not as easy as separating the JPA, disruptor, mongo and couch dependencies.

        OSGi clients have to be explicit about the parts of log4j that they use, so it will be easy to identify the pieces of log4j that need to be installed for a particular client to work, without having to pull in the dependencies for the whole thing.

        You could always use the maven shade plugin to aggregate the individual pieces back into a single JAR for use in Java SE/EE deployments.

        Show
        Timothy Ward added a comment - Remko Popma The simplest way I can think of to achieve #3 would be to split log4j-core into multiple bundles. A "main" bundle containing most of the log4j-core, and then some small extensions for each of the optional pieces. At a glance I would expect there to be an extension for "async", one for JPA, and two NoSQL (one for mongo and one for couch), but there may be others that could be easily extracted to reduce the number of package imports needed. I did wonder whether the jackson dependencies could be easily factored out, but it's not as easy as separating the JPA, disruptor, mongo and couch dependencies. OSGi clients have to be explicit about the parts of log4j that they use, so it will be easy to identify the pieces of log4j that need to be installed for a particular client to work, without having to pull in the dependencies for the whole thing. You could always use the maven shade plugin to aggregate the individual pieces back into a single JAR for use in Java SE/EE deployments.
        Hide
        Ralph Goers added a comment -

        Tim, could you also look at what is being proposed in LOG4J2-238? The patch there seems to make an attempt to fix things but I'm afraid it probably breaks things in a non-OSGi environment.

        How do we break core into multiple bundles without breaking it into multiple Maven projects and multiple jars? Most of us here have very limited OSGi experience so any patches you could provide would be heartily welcomed.

        Show
        Ralph Goers added a comment - Tim, could you also look at what is being proposed in LOG4J2-238 ? The patch there seems to make an attempt to fix things but I'm afraid it probably breaks things in a non-OSGi environment. How do we break core into multiple bundles without breaking it into multiple Maven projects and multiple jars? Most of us here have very limited OSGi experience so any patches you could provide would be heartily welcomed.
        Hide
        Timothy Ward added a comment -

        This patch offers a basic start for making log4j-core more consumable in OSGi.

        Show
        Timothy Ward added a comment - This patch offers a basic start for making log4j-core more consumable in OSGi.
        Hide
        Gary Gregory added a comment -

        -1: Let's not make it a nightmare of a 1000 jars for people who do not live in the OSGi world. please.

        Show
        Gary Gregory added a comment - -1: Let's not make it a nightmare of a 1000 jars for people who do not live in the OSGi world. please.
        Hide
        Remko Popma added a comment - - edited

        Timothy, excuse my osgi ignorance. Do all the bundles become separate jar files?

        Gary, would it help if we zipped up all the bundle jars into a single here-is-your-OSGi-stuff.zip file during the build?

        Show
        Remko Popma added a comment - - edited Timothy, excuse my osgi ignorance. Do all the bundles become separate jar files? Gary, would it help if we zipped up all the bundle jars into a single here-is-your-OSGi-stuff.zip file during the build?
        Hide
        Timothy Ward added a comment -

        Ralph Goers some of the things being proposed in LOG4J2-238 are probably reasonable, although optional imports in OSGi are hard to get right (i.e. avoid NoClassDefFoundError problems). The patch looks as though it's moving in the right direction for better OSGi support, services are a much better mechanism for sharing objects than using fragments and static factories.

        There might be a slightly better approach for structuring the service plug-point so that it's less invasive to the non-OSGi users. I can take a closer look over the next week or so.

        I've attached a bare-bones patch for creating scoped fragments (the minimal change I could see) that should allow OSGi clients to only install the parts of core that they need in a safe way.

        Show
        Timothy Ward added a comment - Ralph Goers some of the things being proposed in LOG4J2-238 are probably reasonable, although optional imports in OSGi are hard to get right (i.e. avoid NoClassDefFoundError problems). The patch looks as though it's moving in the right direction for better OSGi support, services are a much better mechanism for sharing objects than using fragments and static factories. There might be a slightly better approach for structuring the service plug-point so that it's less invasive to the non-OSGi users. I can take a closer look over the next week or so. I've attached a bare-bones patch for creating scoped fragments (the minimal change I could see) that should allow OSGi clients to only install the parts of core that they need in a safe way.
        Hide
        Timothy Ward added a comment - - edited

        Gary Gregory Remko Popma I agree that having to put lots of jars in your application can be a pain, so I deliberately left the original core jar alone. This is the one that you would use in Java SE/EE. The smaller OSGi bundles are typically going to be installed automatically as part of an OSGi provisioning operation. Having separate modules in OSGi is a nicer way of building up function without dragging in things you don't need. Another option is just to mark everything as optional, but this just gives you a "broken out of the box" scenario, where it looks like the bundle is working but actually you really do need some of the dependencies.

        Show
        Timothy Ward added a comment - - edited Gary Gregory Remko Popma I agree that having to put lots of jars in your application can be a pain, so I deliberately left the original core jar alone. This is the one that you would use in Java SE/EE. The smaller OSGi bundles are typically going to be installed automatically as part of an OSGi provisioning operation. Having separate modules in OSGi is a nicer way of building up function without dragging in things you don't need. Another option is just to mark everything as optional, but this just gives you a "broken out of the box" scenario, where it looks like the bundle is working but actually you really do need some of the dependencies.
        Hide
        Gary Gregory added a comment -

        I do not see one solution better than the other: As it is now, I have to manually add a jar for JDBC or JMS, for example: the API jar, Core jar and a JMS provider jar(s) (or a JDBC driver jar(s)). How is having a bunch of little bundles solve this? I still have to manually add jars or bundles for JDBC drivers and JMS providers. The JDBC API is included in the JRE but JMS is not. So even if I have a log4j-jms bundle that drags in the JMS API, that does not help me bring in the JMS provider I want. So the problem cannot be 100% solved with bundles.

        I've never had a Log4J 1 and 2 scenario that did not require fiddling with the classpath, using bundles is not going to change that.

        We should mark our current jars with OSGi optional settings in all the right places, that's a given IMO.

        Show
        Gary Gregory added a comment - I do not see one solution better than the other: As it is now, I have to manually add a jar for JDBC or JMS, for example: the API jar, Core jar and a JMS provider jar(s) (or a JDBC driver jar(s)). How is having a bunch of little bundles solve this? I still have to manually add jars or bundles for JDBC drivers and JMS providers. The JDBC API is included in the JRE but JMS is not. So even if I have a log4j-jms bundle that drags in the JMS API, that does not help me bring in the JMS provider I want. So the problem cannot be 100% solved with bundles. I've never had a Log4J 1 and 2 scenario that did not require fiddling with the classpath, using bundles is not going to change that. We should mark our current jars with OSGi optional settings in all the right places, that's a given IMO.
        Hide
        Timothy Ward added a comment - - edited

        I've never had a Log4J 1 and 2 scenario that did not require fiddling with the classpath, using bundles is not going to change that.

        Actually using bundles is designed to change exactly that. Libraries should be packaged in such a way as to make them as easy to reuse as possible. Ideally they should also be self-describing, which OSGi bundles are. This means that well written OSGi applications set up their own class path without having to be fiddled with.

        To make log4j 2.0 really easy to use in OSGi you don't want users to have to pull in lots of dependencies they don't need, but you also don't want things to look as if they can be used when really they can't. JMS is a good example, the JMS API is one part of the story, but you also need a JMS implementation. What that means is that the bundle has an Import-Package dependency on javax.jms, but also a Generic "Require-Capability" for a JMS provider implementation. You really don't want to pull in the JMS API and an implementation all of that in all of the time if you're only using a FileAppender, so separating it out makes a lot of sense.

        Just making everything optional does avoid the initial "Why do I need all of these dependencies?" problem. But it does so by introducing the "Why doesn't this work?" problem for anyone who does try to use JMS, or Mongo, or any of the other extensions that need the extra dependencies. These dependencies aren't really optional because parts of the log4j API are broken if the dependencies aren't satisfied. If we take the name of this JIRA bug at face value then we should be aiming to make log4j 2.0 work well with OSGi, not attempting to treat OSGi's modularity as something to be worked around

        Show
        Timothy Ward added a comment - - edited I've never had a Log4J 1 and 2 scenario that did not require fiddling with the classpath, using bundles is not going to change that. Actually using bundles is designed to change exactly that. Libraries should be packaged in such a way as to make them as easy to reuse as possible. Ideally they should also be self-describing, which OSGi bundles are. This means that well written OSGi applications set up their own class path without having to be fiddled with. To make log4j 2.0 really easy to use in OSGi you don't want users to have to pull in lots of dependencies they don't need, but you also don't want things to look as if they can be used when really they can't. JMS is a good example, the JMS API is one part of the story, but you also need a JMS implementation. What that means is that the bundle has an Import-Package dependency on javax.jms, but also a Generic "Require-Capability" for a JMS provider implementation. You really don't want to pull in the JMS API and an implementation all of that in all of the time if you're only using a FileAppender, so separating it out makes a lot of sense. Just making everything optional does avoid the initial "Why do I need all of these dependencies?" problem. But it does so by introducing the "Why doesn't this work?" problem for anyone who does try to use JMS, or Mongo, or any of the other extensions that need the extra dependencies. These dependencies aren't really optional because parts of the log4j API are broken if the dependencies aren't satisfied. If we take the name of this JIRA bug at face value then we should be aiming to make log4j 2.0 work well with OSGi, not attempting to treat OSGi's modularity as something to be worked around
        Hide
        Andreas Opitz added a comment -

        You could always use the maven shade plugin to aggregate the individual pieces back into a single JAR for use in Java SE/EE deployments.

        If I understood Timothy Ward correct it should be possible to split the log4j-core into multiple bundles and use the maven shade plugin to aggregate them to a single jar for the none OSGi needs. Is this correct?

        So at the end we would have both multiple bundles for the OSGi needs and also the possibility to great a one jar file.

        Show
        Andreas Opitz added a comment - You could always use the maven shade plugin to aggregate the individual pieces back into a single JAR for use in Java SE/EE deployments. If I understood Timothy Ward correct it should be possible to split the log4j-core into multiple bundles and use the maven shade plugin to aggregate them to a single jar for the none OSGi needs. Is this correct? So at the end we would have both multiple bundles for the OSGi needs and also the possibility to great a one jar file.
        Hide
        Nick Williams added a comment -

        First, I will prefix this with "I know nothing about OSGi." Of course, that's kind of a lie, because I've been able to gleam from this and LOG4J2-238 and discussions on the mailing list that OSGi is some kind of module system for Java. That's all I know.

        Second, I would like to *2 Gary's -1 (i.e., I would like to -1 splitting out the Core into multiple JARs). There are so ... many ... JARs already.

        As I have been the driver and developer of the database Appenders, and since they have been specifically mentioned in these discussions, I will address them and their needs here.

        • Abstract Database Appender (o.a.l.l.core.appender.db): This serves as the base Appender from which all existing database appenders are derived, and it can also be custom-extended by Core users who want to do something different. It depends only on JRE classes and other classes in the Core.
        • JDBC Appender (o.a.l.l.core.appender.db.jdbc): This Appender is for writing events to a relational (SQL) database using JDBC. The only "extra" dependency it has is on the JDBC driver of the user's choice. If I were a betting man and weren't poor, I'd bet bookoos of money that 100% of users of this Appender /already/ are using JDBC for something else and /already/ have the driver on their class path. Problem solved.
        • JPA Appender (o.a.l.l.core.appender.db.jpa): This Appender is for writing events to a relational (SQL) database using JPA 2.1. This has "extra" dependencies of 1) The JPA 2.1 API, 2) The JPA provider of choice, 3) The JDBC driver of choice. The JPA 2.1 API provider is the OSGi dependency that we can export. The JPA provider and JDBC driver are completely up to the user. The /only/ compelling reason to use the JPA Appender instead of the JDBC Appender is if you are already using JPA; otherwise, it's a lot of extra work (read: configuration) for no gain. Therefore, I believe the suggestion that users of this Appender would struggle with NoClassDefFoundErrors if the JPA OSGi dependency were optional is a non-starter. They will already have it on the class path, along with their provider and JDBC driver.
        • NoSQL Appender (o.a.l.l.core.appender.db.nosql): This Appender is for writing events to a non-relational (NoSQL) database. There is no common API like JDBC for using NoSQL databases, since they are all so different. Thus, this appender has no extra dependencies; just the JRE and the Core. It specifies its own common NoSQL API called the NoSQLProvider that specifies only the ability to insert records, because that's all Log4j needs. Users who want to support a currently-unsupported NoSQL database need only implement this simple provider API; they don't have to know anything about Appenders.
        • MongoDB NoSQL Provider (o.a.l.l.core.appender.db.nosql.mongo): This is a MongoDB implementation of the aforementioned NoSQL Provider API. It depends on the Mongo Java Driver supplied by MongoDB. Here is something that could be interesting. I CAN think of two use cases where one might wish to use this provider and not already have the Mongo Java Driver on their class path: 1) In my case, for example, I will be using MongoDB /only/ for logging. I won't use it for any other data storage, I'll be using JPA and MySQL for all my other needs. 2) The Mongo Java Driver is not the only way to access MongoDB from Java, so users might be using MongoDB without using the Mongo Java Driver. There's not an easy answer to this. I still favor it being an optional OSGi dependency with strong documentation.
        • Apache CouchDB NoSQL Provider (o.a.l.l.core.appender.db.nosql.couch): Pretty much ditto the MongoDB provider, but slightly worse. While the Mongo Java Driver is /almost always/ the library of choice for Java users for MongoDB, there are /many/ Java libraries for CouchDB. I had to pick the most popular and "recommended" one: LightCouch. Therefore, the idea that the user may be using a different CouchDB Java library is more likely. There's not an easy answer to this. I still favor it being an optional OSGi dependency with strong documentation.
        Show
        Nick Williams added a comment - First, I will prefix this with "I know nothing about OSGi." Of course, that's kind of a lie, because I've been able to gleam from this and LOG4J2-238 and discussions on the mailing list that OSGi is some kind of module system for Java. That's all I know. Second, I would like to *2 Gary's -1 (i.e., I would like to -1 splitting out the Core into multiple JARs). There are so ... many ... JARs already. As I have been the driver and developer of the database Appenders, and since they have been specifically mentioned in these discussions, I will address them and their needs here. Abstract Database Appender (o.a.l.l.core.appender.db): This serves as the base Appender from which all existing database appenders are derived, and it can also be custom-extended by Core users who want to do something different. It depends only on JRE classes and other classes in the Core. JDBC Appender (o.a.l.l.core.appender.db.jdbc): This Appender is for writing events to a relational (SQL) database using JDBC. The only "extra" dependency it has is on the JDBC driver of the user's choice. If I were a betting man and weren't poor, I'd bet bookoos of money that 100% of users of this Appender /already/ are using JDBC for something else and /already/ have the driver on their class path. Problem solved. JPA Appender (o.a.l.l.core.appender.db.jpa): This Appender is for writing events to a relational (SQL) database using JPA 2.1. This has "extra" dependencies of 1) The JPA 2.1 API, 2) The JPA provider of choice, 3) The JDBC driver of choice. The JPA 2.1 API provider is the OSGi dependency that we can export. The JPA provider and JDBC driver are completely up to the user. The /only/ compelling reason to use the JPA Appender instead of the JDBC Appender is if you are already using JPA; otherwise, it's a lot of extra work (read: configuration) for no gain. Therefore, I believe the suggestion that users of this Appender would struggle with NoClassDefFoundErrors if the JPA OSGi dependency were optional is a non-starter. They will already have it on the class path, along with their provider and JDBC driver. NoSQL Appender (o.a.l.l.core.appender.db.nosql): This Appender is for writing events to a non-relational (NoSQL) database. There is no common API like JDBC for using NoSQL databases, since they are all so different. Thus, this appender has no extra dependencies; just the JRE and the Core. It specifies its own common NoSQL API called the NoSQLProvider that specifies only the ability to insert records, because that's all Log4j needs. Users who want to support a currently-unsupported NoSQL database need only implement this simple provider API; they don't have to know anything about Appenders. MongoDB NoSQL Provider (o.a.l.l.core.appender.db.nosql.mongo): This is a MongoDB implementation of the aforementioned NoSQL Provider API. It depends on the Mongo Java Driver supplied by MongoDB. Here is something that could be interesting. I CAN think of two use cases where one might wish to use this provider and not already have the Mongo Java Driver on their class path: 1) In my case, for example, I will be using MongoDB /only/ for logging. I won't use it for any other data storage, I'll be using JPA and MySQL for all my other needs. 2) The Mongo Java Driver is not the only way to access MongoDB from Java, so users might be using MongoDB without using the Mongo Java Driver. There's not an easy answer to this. I still favor it being an optional OSGi dependency with strong documentation. Apache CouchDB NoSQL Provider (o.a.l.l.core.appender.db.nosql.couch): Pretty much ditto the MongoDB provider, but slightly worse. While the Mongo Java Driver is /almost always/ the library of choice for Java users for MongoDB, there are /many/ Java libraries for CouchDB. I had to pick the most popular and "recommended" one: LightCouch. Therefore, the idea that the user may be using a different CouchDB Java library is more likely. There's not an easy answer to this. I still favor it being an optional OSGi dependency with strong documentation.
        Hide
        Timothy Ward added a comment -

        I would like to emphasize that I am not proposing that the existing log4j-core.jar cease to be. I completely understand the desire to have a single consumable unit for Java SE and Java EE to make things as simple as possible. This can (and should) remain the output of log4j-core.

        For OSGi clients having multiple bundles is much less difficult than having lots of extra, potentially optional, dependencies (almost exactly the opposite to Java SE). This means that libraries like log4j end up with three main options when creating OSGi bundles:

        1. Simply use the maven bundle plugin to add OSGi metadata and declare victory. This leads to issues like LOG4J2-238 where OSGi clients are presented with a large number of opaque dependencies. Most of which are completely unnecessary but have to be provided. As an example, to use any functions from log4j-core (even the simplest appender) in OSGi right now I would have to install a mongo driver, a couch driver, the JMS api, the JPA api and a whole heap of other stuff. This effectively makes log4j-core completely unusable for OSGi clients

        2. Use the maven bundle plugin to add OSGi metadata, but declare all of the imports as optional (possibly excluding org.apache.logging.log4j.). This means that the log4j-core bundle will always resolve, however it is still very unpleasant to use. Once resolved the log4j-core bundle exposes *all of its exported packages (e.g. o.a.l.l.core.appender.db.nosql.mongo and o.a.l.l.core.appender.db.nosql.couch). These packages will definitely not work unless the relevant dependencies are available, mostly they will throw ClassNotFoundException and NoClassDefFoundError at unpleasant points. The end result is a lot of broken OSGi bundles trying to use features that aren't actually available.

        3. Modularise the component into OSGi bundles grouped by function and dependency. This can either be done by splitting the source into multiple maven modules (the aggregate jar can be recreated using the shade plugin), or by using the maven bundle plugin to create bundles from a single source jar (which is what the attached patch does). OSGi clients can then install the parts of log4j that they actually need, for example a simple client can just install the core-osgi-reduced bundle, but one that wants to use mongo can add the core-osgi-nosql-mongo bundle and the mongo driver bundle.

        The OSGi bundles created by the patch are as follows:

        a) core-osgi-reduced - this is most of the log4j-core jar, with as many external (non o.a.l.l) dependencies removed as possible. Once LOG4J2-238 is addressed this should run using only JDK provided code and have no external dependencies

        b) core-osgi-async - this pulls out the o.a.l.l.core.async package to separate the dependency on lmax disruptors. If LOG4J2-238 can't be addressed then this could be pushed back into a)

        c) core-osgi-net - this pulls out the o.a.l.l.core.net package to separate the javax.jms, javax.mail and javax.activation dependencies

        d) core-osgi-nosql-couch - this pulls out the dependency on LightCouch

        e) core-osgi-nosql-mongo - this pulls out the dependency on Mongo

        I understand that there's a strong temptation to say "Let's just make X an optional dependency", however in OSGi an optional dependency really is optional, i.e. all the functions of the bundle should continue to work, possibly in a slower or slightly restricted manner. In log4j the "optional" packages would really indicate that some of the exported packages do not work at all. This can and does break OSGi applications.

        Show
        Timothy Ward added a comment - I would like to emphasize that I am not proposing that the existing log4j-core.jar cease to be. I completely understand the desire to have a single consumable unit for Java SE and Java EE to make things as simple as possible. This can (and should) remain the output of log4j-core. For OSGi clients having multiple bundles is much less difficult than having lots of extra, potentially optional, dependencies (almost exactly the opposite to Java SE). This means that libraries like log4j end up with three main options when creating OSGi bundles: 1. Simply use the maven bundle plugin to add OSGi metadata and declare victory. This leads to issues like LOG4J2-238 where OSGi clients are presented with a large number of opaque dependencies. Most of which are completely unnecessary but have to be provided. As an example, to use any functions from log4j-core (even the simplest appender) in OSGi right now I would have to install a mongo driver, a couch driver, the JMS api, the JPA api and a whole heap of other stuff. This effectively makes log4j-core completely unusable for OSGi clients 2. Use the maven bundle plugin to add OSGi metadata, but declare all of the imports as optional (possibly excluding org.apache.logging.log4j. ). This means that the log4j-core bundle will always resolve, however it is still very unpleasant to use. Once resolved the log4j-core bundle exposes *all of its exported packages (e.g. o.a.l.l.core.appender.db.nosql.mongo and o.a.l.l.core.appender.db.nosql.couch). These packages will definitely not work unless the relevant dependencies are available, mostly they will throw ClassNotFoundException and NoClassDefFoundError at unpleasant points. The end result is a lot of broken OSGi bundles trying to use features that aren't actually available. 3. Modularise the component into OSGi bundles grouped by function and dependency. This can either be done by splitting the source into multiple maven modules (the aggregate jar can be recreated using the shade plugin), or by using the maven bundle plugin to create bundles from a single source jar (which is what the attached patch does). OSGi clients can then install the parts of log4j that they actually need, for example a simple client can just install the core-osgi-reduced bundle, but one that wants to use mongo can add the core-osgi-nosql-mongo bundle and the mongo driver bundle. The OSGi bundles created by the patch are as follows: a) core-osgi-reduced - this is most of the log4j-core jar, with as many external (non o.a.l.l) dependencies removed as possible. Once LOG4J2-238 is addressed this should run using only JDK provided code and have no external dependencies b) core-osgi-async - this pulls out the o.a.l.l.core.async package to separate the dependency on lmax disruptors. If LOG4J2-238 can't be addressed then this could be pushed back into a) c) core-osgi-net - this pulls out the o.a.l.l.core.net package to separate the javax.jms, javax.mail and javax.activation dependencies d) core-osgi-nosql-couch - this pulls out the dependency on LightCouch e) core-osgi-nosql-mongo - this pulls out the dependency on Mongo I understand that there's a strong temptation to say "Let's just make X an optional dependency", however in OSGi an optional dependency really is optional, i.e. all the functions of the bundle should continue to work, possibly in a slower or slightly restricted manner. In log4j the "optional" packages would really indicate that some of the exported packages do not work at all. This can and does break OSGi applications.
        Hide
        Ralph Goers added a comment -

        Thanks for the explanation. I am strongly in favor of this approach but I do have a question about the names. First, I would prefer that all of these reside under the "org.apache.logging.log4j.osgi" groupId to separate them from the non-osgi artifacts, but I'm not sure if that is possible with the maven bundle plugin. If we do that would the "osgi" in the artifact names still be required? I assume that all of these are still jars so I suppose users might get confused. In addition, these are all going to appear in the distribution archive. Ideally they would be separate from the other jars.

        I'm going to apply your patch when I can get a chance and I'm sure I will have more questions.

        Show
        Ralph Goers added a comment - Thanks for the explanation. I am strongly in favor of this approach but I do have a question about the names. First, I would prefer that all of these reside under the "org.apache.logging.log4j.osgi" groupId to separate them from the non-osgi artifacts, but I'm not sure if that is possible with the maven bundle plugin. If we do that would the " osgi " in the artifact names still be required? I assume that all of these are still jars so I suppose users might get confused. In addition, these are all going to appear in the distribution archive. Ideally they would be separate from the other jars. I'm going to apply your patch when I can get a chance and I'm sure I will have more questions.
        Hide
        Nick Williams added a comment -

        I'm okay with using the Maven bundle plugin to create bundles from a single source. I would strongly oppose any move to split the source into multiple Maven modules and aggregate the JAR (this would get very ugly quickly; note that we wouldn't need the shade plugin, the assembly plugin would be sufficient).

        Note that there will also need to be a core-osgi-jpa (or equivalent) bundle. You did not list that one, and I don't know if it's because you forgot to mention it or it's missing from your patch. There must be a bundle that pulls out the dependency on JPA.

        Show
        Nick Williams added a comment - I'm okay with using the Maven bundle plugin to create bundles from a single source. I would strongly oppose any move to split the source into multiple Maven modules and aggregate the JAR (this would get very ugly quickly; note that we wouldn't need the shade plugin, the assembly plugin would be sufficient). Note that there will also need to be a core-osgi-jpa (or equivalent) bundle. You did not list that one, and I don't know if it's because you forgot to mention it or it's missing from your patch. There must be a bundle that pulls out the dependency on JPA.
        Hide
        Timothy Ward added a comment -

        A couple of things were pointed out:

        1. It would be better if the additional artifacts added osgi to their groupId - this is changed (and easy to change back)

        2. The first patch missed creating a core-osgi-jpa bundle - this is fixed

        Currently the core-osgi-reduced bundle has external dependencies on com.lmax.disruptor.util and a couple of Jackson packages. Removing these dependencies will require code changes (either to remove the dependency, or to push it into a package that can be extracted into a separate bundle)

        The symbolic names of these bundles are still $

        {project.parent.groupId}

        .$

        {project.artifactId}

        which means they are currently of the form:

        o.a.l.log4j-core-osgi-XXXX

        This is easy to change, but care should be taken to avoid clashes as OSGi uses symbolic name and version as a unique identifier for the bundle.

        Show
        Timothy Ward added a comment - A couple of things were pointed out: 1. It would be better if the additional artifacts added osgi to their groupId - this is changed (and easy to change back) 2. The first patch missed creating a core-osgi-jpa bundle - this is fixed Currently the core-osgi-reduced bundle has external dependencies on com.lmax.disruptor.util and a couple of Jackson packages. Removing these dependencies will require code changes (either to remove the dependency, or to push it into a package that can be extracted into a separate bundle) The symbolic names of these bundles are still $ {project.parent.groupId} .$ {project.artifactId} which means they are currently of the form: o.a.l.log4j-core-osgi-XXXX This is easy to change, but care should be taken to avoid clashes as OSGi uses symbolic name and version as a unique identifier for the bundle.
        Hide
        Nick Williams added a comment -

        I need to point out a couple more things :

        1) I'm not sure exactly what it does, but assuming the _removeheaders pattern is guaranteed consistent, it needs to also include JAVA_1_8_HOME (already lots of installations out there) and JAVA_1_9_HOME (people will start working on this soon). This needs to be changed elsewhere in the codebase, too, but that's a different issue.

        2) I did forget one (tricky) dependency for core-osgi-jpa. Two classes, o.a.l.l.core.appender.db.jpa.converter.ContextMapJsonAttributeConverter and ContextStackJsonAttributeConverter, use Jackson Mapper. Now, if you use the JPA Appender out of the box, it doesn't use these. It uses the non-JSON equivalent ContextMapAttributeConverter and ContextStackAttributeConverter. You have to literally write code to override the mappings to tell it to use the JSON versions. So I think it's safe for Jackson Mapper to be an optional dependency here, but I wanted to make sure you knew.

        Also, a question:

        Since all of the changes in this patch technically add additional Maven submodules, does this affect the site? I don't think we want six extra components to show up on the site. Have you `mvn site` to check on this? The Maven site plugin needs to skip these altogether.

        Show
        Nick Williams added a comment - I need to point out a couple more things : 1) I'm not sure exactly what it does, but assuming the _removeheaders pattern is guaranteed consistent, it needs to also include JAVA_1_8_HOME (already lots of installations out there) and JAVA_1_9_HOME (people will start working on this soon). This needs to be changed elsewhere in the codebase, too, but that's a different issue. 2) I did forget one (tricky) dependency for core-osgi-jpa. Two classes, o.a.l.l.core.appender.db.jpa.converter.ContextMapJsonAttributeConverter and ContextStackJsonAttributeConverter, use Jackson Mapper. Now, if you use the JPA Appender out of the box, it doesn't use these. It uses the non-JSON equivalent ContextMapAttributeConverter and ContextStackAttributeConverter. You have to literally write code to override the mappings to tell it to use the JSON versions. So I think it's safe for Jackson Mapper to be an optional dependency here, but I wanted to make sure you knew. Also, a question: Since all of the changes in this patch technically add additional Maven submodules, does this affect the site? I don't think we want six extra components to show up on the site. Have you `mvn site` to check on this? The Maven site plugin needs to skip these altogether.
        Hide
        Ralph Goers added a comment -

        I'm prepared to commit the patch (with a few modifications) but I am wondering how to test this. There aren't any unit tests in the patch so how can we verify that it works?

        Show
        Ralph Goers added a comment - I'm prepared to commit the patch (with a few modifications) but I am wondering how to test this. There aren't any unit tests in the patch so how can we verify that it works?
        Hide
        Remko Popma added a comment -

        Ralph, did you eventually commit this?
        If so, Timothy, could you give us feedback if this fulfills your expectations?

        Show
        Remko Popma added a comment - Ralph, did you eventually commit this? If so, Timothy, could you give us feedback if this fulfills your expectations?
        Hide
        Ralph Goers added a comment -

        Yes. But I still have no idea how to test it.

        Show
        Ralph Goers added a comment - Yes. But I still have no idea how to test it.
        Hide
        Timothy Ward added a comment -

        Hi Ralph,

        The bundles are looking pretty good - there's still a little tweaking needed for the build classpath, at the moment there are no version ranges on the OSGi framework imports, nor for Jackson. This should just be a matter of making sure the relevant bundles are declared as maven dependencies.

        For testing, I suggest you look at PAX Exam. This is what Apache Aries uses for OSGi functional tests.

        Documentation for PAX exam is available here:

        https://ops4j1.jira.com/wiki/display/PAXEXAM3/Getting+Started+with+OSGi+Tests

        We should be able to write simple sniff tests for the various appenders that are very similar to their existing unit tests. This will allow us to verify that all of them work in OSGi.

        Show
        Timothy Ward added a comment - Hi Ralph, The bundles are looking pretty good - there's still a little tweaking needed for the build classpath, at the moment there are no version ranges on the OSGi framework imports, nor for Jackson. This should just be a matter of making sure the relevant bundles are declared as maven dependencies. For testing, I suggest you look at PAX Exam. This is what Apache Aries uses for OSGi functional tests. Documentation for PAX exam is available here: https://ops4j1.jira.com/wiki/display/PAXEXAM3/Getting+Started+with+OSGi+Tests We should be able to write simple sniff tests for the various appenders that are very similar to their existing unit tests. This will allow us to verify that all of them work in OSGi.
        Hide
        Nick Williams added a comment -

        Tim,

        One change that has been made recently is that classes in o.a.l.l.core.web in the Core component reference the Servlet API (these classes used to be in their own component). One of these classes requires Servlet 3.0 and the other classes require Servlet 2.5. NONE of the classes are referenced by any classes outside of that package, so I don't THINK it's going to cause an OSGi problem. These classes are ONLY used through scanning (if you're in a Servlet 3.0 environment already, so the Servlet classes are already on the classpath) or through the deployment descriptor (if you're in a Servlet 2.5 environment, ditto).

        The purpose of this change was to keep Web application users from having to include an additional JAR just to get the Core to initialize/deinitialize correctly in a Web application. What do you think? Is this going to be a problem OSGi-wise?

        Nick

        Show
        Nick Williams added a comment - Tim, One change that has been made recently is that classes in o.a.l.l.core.web in the Core component reference the Servlet API (these classes used to be in their own component). One of these classes requires Servlet 3.0 and the other classes require Servlet 2.5. NONE of the classes are referenced by any classes outside of that package, so I don't THINK it's going to cause an OSGi problem. These classes are ONLY used through scanning (if you're in a Servlet 3.0 environment already, so the Servlet classes are already on the classpath) or through the deployment descriptor (if you're in a Servlet 2.5 environment, ditto). The purpose of this change was to keep Web application users from having to include an additional JAR just to get the Core to initialize/deinitialize correctly in a Web application. What do you think? Is this going to be a problem OSGi-wise? Nick
        Hide
        Timothy Ward added a comment -

        Hi Nick,

        You definitely want to declare the import for the servlet package so that web bundles can use log4j without having to put it in WEB-INF/lib. This means that you also need to export the o.a.l.l.core.web package (so that web bundles can import it and use it). The maven bundle plugin will calculate your uses constraint to make sure this all works properly.

        If o.a.l.l.core.web works with Servlet 2.5, then the OSGi packaging should use servlet 2.5 on the build path to pick up the right import version range. This should be "[2.5,3)" because servlet 3 has a semantic version of 2.6

        From your description of how things are laid out I don't forsee any problems in OSGi, assuming that the Import-Package version range is set properly.

        Tim

        Show
        Timothy Ward added a comment - Hi Nick, You definitely want to declare the import for the servlet package so that web bundles can use log4j without having to put it in WEB-INF/lib. This means that you also need to export the o.a.l.l.core.web package (so that web bundles can import it and use it). The maven bundle plugin will calculate your uses constraint to make sure this all works properly. If o.a.l.l.core.web works with Servlet 2.5, then the OSGi packaging should use servlet 2.5 on the build path to pick up the right import version range. This should be "[2.5,3)" because servlet 3 has a semantic version of 2.6 From your description of how things are laid out I don't forsee any problems in OSGi, assuming that the Import-Package version range is set properly. Tim
        Hide
        Nick Williams added a comment -

        So, OSGi already sounds far more complicated than anything I will ever want to use. I'm starting to think of "DLL Hell." Regular JAR stuff "just works."

        So o.a.l.l.core.web works with Servlet 2.5, 3.0, 3.1, etc. So it needs to be "2.5+" however that works.

        Log4j would normally go in /WEB-INF/lib. It wouldn't normally go in the container unless the container is actually using Log4j. Even then, Log4j could still go in /WEB-INF/lib. But I don't understand any of this.

        Everything you just said is complete gibberish to me, so somebody else is going to have to make whatever changes need making.

        Nick

        Show
        Nick Williams added a comment - So, OSGi already sounds far more complicated than anything I will ever want to use. I'm starting to think of "DLL Hell." Regular JAR stuff "just works." So o.a.l.l.core.web works with Servlet 2.5, 3.0, 3.1, etc. So it needs to be "2.5+" however that works. Log4j would normally go in /WEB-INF/lib. It wouldn't normally go in the container unless the container is actually using Log4j. Even then, Log4j could still go in /WEB-INF/lib. But I don't understand any of this. Everything you just said is complete gibberish to me, so somebody else is going to have to make whatever changes need making. Nick
        Hide
        Pedro Lamarão added a comment -

        Regular JAR stuff doesn't just work.
        It leads to "JAR hell".
        You won't see "JAR Hell" between two servlet modules because they live not under regular JAR rules, they live under J2EE module isolation rules.
        In particular, servlet modules have this WEB-INF/lib trick where every dependency is "private".
        This is equivalent to the solution Microsoft gave to "DLL Hell" – currently named "private assemblies".

        Of course, in practice, everything in a servlet container ends up being "private" somehow.
        "shared" just doensn't work.
        Because "shared" leads to "JAR Hell".

        The frustration caused by OSGi's strictness is understandable.
        But OSGi is a solution to a true problem – which you may or may not have.
        I know I do.

        Show
        Pedro Lamarão added a comment - Regular JAR stuff doesn't just work. It leads to "JAR hell". You won't see "JAR Hell" between two servlet modules because they live not under regular JAR rules, they live under J2EE module isolation rules. In particular, servlet modules have this WEB-INF/lib trick where every dependency is "private". This is equivalent to the solution Microsoft gave to "DLL Hell" – currently named "private assemblies". Of course, in practice, everything in a servlet container ends up being "private" somehow. "shared" just doensn't work. Because "shared" leads to "JAR Hell". The frustration caused by OSGi's strictness is understandable. But OSGi is a solution to a true problem – which you may or may not have. I know I do.
        Hide
        Remko Popma added a comment -

        Timothy, we may not have the expertise in the Log4j team to make the changes you mention. Would you mind submitting a patch? Many thanks in advance!

        Show
        Remko Popma added a comment - Timothy, we may not have the expertise in the Log4j team to make the changes you mention. Would you mind submitting a patch? Many thanks in advance!
        Hide
        Timothy Ward added a comment -

        Hi,

        As requested I've worked on a patch which improves the version ranges on the various log4j2 bundles, and also that adds a couple of simple OSGi sniff tests. Hopefully these can be used as a base to build a more comprehensive set of tests for the various appenders, but at the very least they prove that the bundles can be used successfully.

        In the patch are the following changes:

        Updates to various pom files. This corrects the version ranges and adds the maven dependencies for PAX Exam.

        A new project for core-osgi-web, allowing the javax.servlet dependency to be separated from the dependency reduced core.

        A new project for the OSGi sniff tests. This uses PAX Exam to launch an OSGi framework, and runs a couple of simple logging tests, once with the full core bundle, and once with the dependency reduced core bundle. This did flush out a couple of issues, fixes for which are included in the patch, and outlined below.

        o.a.l.l.util.ProviderUtil - the change here is necessary to prevent the ThreadContextClassLoader (which is frequently the bootstrap classloader) from hiding the log4j implementation. Without this it is impossible to load the configuration or Classes for Log4J. The TCCL still takes precedence when loading things (which is definitely the right behaviour for Web Apps) but if nothing is found then the code falls back to using the Log4J classloader.

        o.a.l.l.core.config.LoggerConfig - this class was causing a cyclic package dependency on o.a.l.l.core.async and therefore causing problems in the dependency reduced bundle. As the only use of this package was to get the name of a class I simply replaced it with a literal. The class name was part of a client-set property, and so shouldn't ever change.

        o.a.l.l.core.config.plugins - There is another cyclic package dependency between o.a.l.l.core and o.a.l.l.core.net. This is not possible to resolve without moving classes, so I have worked around it. This means that three classes from o.a.l.l.core.net are included in the dependency reduced bundle. As a result most of the classes in the dependency reduced bundle (e.g. the syslog layout) are usable, but some aren't (e.g. the JMSTopicAppender). These classes do work if you install the osgi-core-net fragment as well as the dependency reduced bundle, but in cases where you haven't we need to handle the ClassNotFound/NoClassDefFound. This has the side benefit of making those plugins unavailable unless their dependencies are satisfied.

        Show
        Timothy Ward added a comment - Hi, As requested I've worked on a patch which improves the version ranges on the various log4j2 bundles, and also that adds a couple of simple OSGi sniff tests. Hopefully these can be used as a base to build a more comprehensive set of tests for the various appenders, but at the very least they prove that the bundles can be used successfully. In the patch are the following changes: Updates to various pom files. This corrects the version ranges and adds the maven dependencies for PAX Exam. A new project for core-osgi-web, allowing the javax.servlet dependency to be separated from the dependency reduced core. A new project for the OSGi sniff tests. This uses PAX Exam to launch an OSGi framework, and runs a couple of simple logging tests, once with the full core bundle, and once with the dependency reduced core bundle. This did flush out a couple of issues, fixes for which are included in the patch, and outlined below. o.a.l.l.util.ProviderUtil - the change here is necessary to prevent the ThreadContextClassLoader (which is frequently the bootstrap classloader) from hiding the log4j implementation. Without this it is impossible to load the configuration or Classes for Log4J. The TCCL still takes precedence when loading things (which is definitely the right behaviour for Web Apps) but if nothing is found then the code falls back to using the Log4J classloader. o.a.l.l.core.config.LoggerConfig - this class was causing a cyclic package dependency on o.a.l.l.core.async and therefore causing problems in the dependency reduced bundle. As the only use of this package was to get the name of a class I simply replaced it with a literal. The class name was part of a client-set property, and so shouldn't ever change. o.a.l.l.core.config.plugins - There is another cyclic package dependency between o.a.l.l.core and o.a.l.l.core.net. This is not possible to resolve without moving classes, so I have worked around it. This means that three classes from o.a.l.l.core.net are included in the dependency reduced bundle. As a result most of the classes in the dependency reduced bundle (e.g. the syslog layout) are usable, but some aren't (e.g. the JMSTopicAppender). These classes do work if you install the osgi-core-net fragment as well as the dependency reduced bundle, but in cases where you haven't we need to handle the ClassNotFound/NoClassDefFound. This has the side benefit of making those plugins unavailable unless their dependencies are satisfied.

          People

          • Assignee:
            Unassigned
            Reporter:
            Curt Arnold
          • Votes:
            2 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:

              Development