|
I'm finally getting back to this exercise and I will have to admit that this is quite difficult not understanding the general architecture of thse ConfigurationProviders and ProductDerivations. Instead of jumping right in and attempting to make Abe's suggested changes, I'm trying to understand the current architecture. Here's what I am finding...
The application attempts to create an EMF via Persistence.createEntityManagerFactory(). This eventually gets us to the PersistenceProviderImpl.createEntityManagerFactory() methods, which creates the ConfigurationProviderImpl instance. We then use this ConfigurationProviderImpl instance to load the resources and create the PersistenceUnitInfo object. The last thing we do in createEntityManagerFactory is to create the appropriate BrokerFactory. We do this by calling Bootstrap.newBrokerFactory passing in the ConfigurationProvider instance and associated Loader. Within this processing, I see where we are looping through the ProductDerivation services (sorted on Type). This, in turn, creates the new JDBCBrokerFactory instance, which creates the corresponding JDBCConfigurationImpl. After initializing the properties in the constructor, we attempt to load the global settings by calling Configurations.loadGlobals static method (via the ConfigurationImpl.loadGlobals method). This is where it gets confusing. Within this method, we create new instances of the ConfigurationProviderImpl and we use these instances to load resources (again?). So, I am seeing where we loop through ConfigurationProviders, eventually creating and looping thorugh ProductDerivations, and then we create and process new instances of ConfigurationProviders. Granted, I've only spent a couple of hours looking at this today, but I think I need some background on how these Services were working previously before attempting to correct it. Maybe this was part of the reason why Abe wanted to get rid of the ConfigurationProvider service and drive everything through the ProductDerivations. It's kind of confusing. Thanks, Keivn Globals are configuration properties that are always loaded before any explicit configuration supplied by the user, both when using dev tools and at runtime. Each configuration provider decides on whether and how it uses globals. The JPA configuration provider does not use globals. Kodo's JDO configuration provider looks for a "kodo.properties" resource in the classpath and loads anything in it. A Kodo user could theoretically have a kodo.properties with JDO properties, but request an EntityManagerFactory through Persistence.
Thus the loading of globals is independent of the runtime ConfigurationProvider created by PersistenceProviderImpl or any other bootstrapping mechanism. That is reflected in the behavior Kevin noted: when loading globals we loop through all the ConfigurationProviders in the system until we find one that loads successfully, regardless of how the user is bootstrapping his runtime. Correction: the JPA configuration provider does use globals. It looks for an openjpa.xml resource for global configuration. Just as a Kodo user could have a kodo.properties but bootstrap his runtime through JPA, he could also use openjpa.xml but bootstrap his runtime through JDO. Again, the loading of globals is independent of the runtime bootstrapping mechanism.
Just to get it right too. You can bootstrap OpenJPA by the means of PersistenceProviderImpl but still obtain a JDO persistenceManager. But the important question is, where to put things which were formerly in kodo.properties? Or is PersistenceProviderImpl able to load from kodo.properties?
One use case for extension of OpenJPA implementations is backward compatibility of Kodo 4.1 with Kodo 4.0. In Kodo 4.0, we have published interfaces such as kodo.persistence.KodoEntityManagerFactory/KodoEntityManager. The applications compiled with Kodo 4.0 API to work with Kodo 4.1 runtime (based on OpenJPA), we would be supporting KodoEntityManagerFactory and other published interfaces.
The published Kodo 4.0 interfaces would be redefined to extend openjpa interfaces for Kodo 4.1. I'm working on some related code for backwards-compatibility of Kodo, which turns out to be a similar problem to extensibility of OpenJPA. Let's compare notes, and see where we're overlapping and how we can help each other out. Concerning the ProductDerivation types...
Of the types defined in ProductDerivation, it looks like only TYPE_SPEC, TYPE_STORE, and TYPE_SPEC_STORE are being used. (I don't find any references to TYPE_PRODUCT, TYPE_PRODUCT_STORE, or TYPE_FEATURE. Must be for future extensions?) I'm assuming that any re-factoring of these types should continue to include these types that are not currently being utilized. One of Abe's earlier comments indicated that if we move ProductDerivation into lib, then we should remove the concept of SPEC and STORE from that interface since lib is persistence-neutral. (Later on, Abe indicated that maybe we could leave the concept of SPEC since that is pretty general, but STORE is definitely specific to persistence.) These removed concepts needed to be re-introduced into the kernel, possibly as a derived OpenJPAProductDerivation. This would imply that the getType() method and the associated constants for the TYPE_* values should be removed from the ProductDerivation interface. But, if we go that route, then we're screwed with our proposed looping through the list of ProductDerivations since it relies on the ProductDerivation.getType() method. So, it would seem that we still need the getType() method and associated TYPE_* constants at the ProductDerivation interface. It seems that it would be okay for the interface to define the various types, and let the implementations deal with the SPEC and/or STORE implications. What am I missing? Kevin The ProductDerivation interface can have a getType() and can order on that type without having to define all the type constants itself. My usual strategy is to define the constants that make sense for the base type, leaving big "holes" so that subclasses can insert their own constants for proper ordering:
public interface ProductDerivation { public static final int BASETYPE1 = 10; public static final int BASETYPE2 = 20; ... } It's somewhat delicate (I usually Javadoc the base constants with their values and the fact that they shouldn't be changed), but it's simple and IMO it's better than the base interface containing a bunch of meaningless (to it) constants. It looks like Pinaki has dropped some code via revision r447664 that provides at least some of the infrastructure needed for this JIRA report. I will need to compare his changes with the changes I was experimenting with to see if this report is now resolved or not. Any additional changes will be logged via this JIRA report.
I had made certain changes for extending OpenJPA with alternative implementaions via ProductDerivations mechanics. Let me present my understanding of this issue raised orginally by Kevin on this discussion thread.
org.apache.openjpa.lib.conf.Configuration carries the properties that determines configurable behaviour e.g. which concrete PersistenceProvider to create or whether to synchronize the object schema with that of the database or whether to apply JPA or JDO style mapping primitives and so on. It is a rich and powerful construct with support for Plugin. The purpose of OpenJPA configuration subsystem is to create one such Configuration instance that is essential for any particular instantiation of the generic kernel known as BrokerFactory. In fact, BrokerFactory and Configuration enjoy a 1:1 till-detah-do-us-apart sort of relationship. ProductDerivation, ProductDerivations, ConfigurationProvider, Configurations are abstractions that participate in creating this Configuration instance. Let us see the roles played by each of them. ConfigurationProvider locates where the configuration information is and then reads it. The information can be in a META-INF/persistence.xml file inside a jar, a kodo.properties file available in classpath, a -Dxyz=myValue style Java system property, in a Map instance constructed programatically - the possibilities are not constrained by design. Because OpenJPA -- notwithstanding its name -- is by design capable of supporting multiple specifications that differs in configuration grammar -- multiple ConfigurationProvider classes are provided. Given the varied nature of how configuration information can be made available to the runtime, the basic interface org.apache.openjpa.lib.conf.ConfigurationProvider provides the discipline of reading configuration from 'global' or 'default' or named resources. Of course, each concrete implementation would interpret what 'global' or 'default' would mean. ConfigurationProvider after locating the information resource, reads its content and temporarilly stores in an internal name-value map. Eventually it pours this content into a Configuration instance via ConfigurationProvider.setInto(Configuration conf) method. ProductDerivation faciliates how Configuration will deal with this content -- which MetaDataFactory to set, which EntityManagerFactory (or PersistenceManagerFactory) to instantiate as a facade to the kernel according to active specification and so on. This tunning is accomplished by hooks during the life of a Configuration before being put to active duty i.e. before a Configuration instance is constructed, before the content carried by ConfigurationProvider is poured in Configuration and after a Configuration is set to represent a specification. For example, the spec-agnostic core configuration implementation ConfigurationImpl nor its derivation OpenJPAConfigurationImpl declares a plugin for which concrete EntityManagerFactory to construct as a facade. But PersistenceProductDerivation inserts org.apache.openjpa.persistence.EntityManagerFactoryImpl as the concrete implementation class for EMF in beforeConfigurationConstruct() hook and adds a EMF-plugin value via beforeConfigurationLoaded() hook i.e. before ConfigurationProvider pours its content into a Configuration. This allows the PersistenceProviderImpl to instantiate a org.apache.openjpa.persistence.EntityManagerFactoryImpl as a facade to BrokerFactory. Given that OpenJPA supports an extensive set of configurable parameters it is logical to separate them into categories such as SPEC, PRODUCT, STORE etc -- and that lead to a host of ProductDerivation classes each tunning the configuration from its own perspective. org.apache.openjpa.lib.conf.ProductDerivations is the harness that locates each ProductDerivation available to the system, order them up sequentially to give a chance to modify Configuration/ConfigurationProvider. ProductDerivations finds ProductDerivation by looking up one or more "org.apache.openjpa.lib.conf.ProductDerivation" resources in the classpath and interpreting each line of this simple text-based resource as a class name for a particular org.apache.openjpa.lib.conf.ProductDerivation implementation. Configurations hold a bunch of static utility methods to instantiate plugin, pour system properties into Configuration and so on. I have not looked into this class due dilligence and it may even be a candidate for being refactored out completely later. Given this scheme, the most visible (and mechanical) change is to drive the loading of configuration data by the ConfigurationProvider via ProductDerivations. It used to be such that different ConfigurationProvider were activated by Configurations and different ProductDerivation were activated by ProductDerivations. Now ProductDerivations is the only driver of configuration subsystem. Each ProductDerivation can supply its own ConfigurationProvider to locate/parse/read configuration information and supplying a null imply that this ProductDerivation does not read resource at all. In fact, most of them don't. This ProductDerivation-as-driver-of-ConfigurationProvider notion is coded into AbstractProductDerivation. The other change as outlined by Abe is to move ProductDerivations/ProductDerivation/Configuration to lib and factor out STORE specifc details in kerenl.OpenJPAProductDerivation. With all these machinery and refactoring -- now let us go back to the issue Kevin originally raised -- how does one extend OpenJPA? The use case became real when we needed a backward compatibility support for Kodo 4.0. Kodo 4.0 was released few months ago in pre-OpenJPA era. Obviously, a mechanism is needed such that applications written on Kodo 4.0 but running on Kodo 4.1 based on OpenJPA must be able to use the old API of kodo.persistence.PersistenceProviderImpl instead of org.apache.... In my next post, I will describe how that was done with ProductDerivation, I have to now attend to booth duty at BEAWorld. It looks like creating an extension is fairly straight forward. With the changes Pinaki committed yesterday I was able to create a simple test extension. All I did was create three classes, a ProductDerivation, a PersistenceProvider, and a ConfigurationProvider. I extended PersistenceProductDerivation, PersistenceProviderImpl and ConfigurationImpl respectively, although I suppose one could write their own if they really wanted/needed to.
To get a fairly simple wrapper extension working all I had to do was override : ProductDerivation.newConfigurationProvider() , and ConfigurationProvider.getPersistenceProviderName(). I didn't see any methods in PersistenceProviderImpl that I needed to change. The last thing I did was register the ProductDerivation and PersistenceProvider as services, and update persistence.xml (specifying the new PersistenceProvider for my persistent unit). After that the new PersistenceProvider, etc. were used and generated the configuration appropriately (as far as I can tell). Each ProductDerivation is called to load a ConfigurationProvider until the first non-null ConfigurationProvider is found. The ConfigProvider and PersistenceProvider need to match and so forth. Is there anything glaring that I missed? Admittedly all I did was a proof of concept. For a real world extension like Kodo 4.1 there will be other bits to change, changing the default properties in ConfigurationProvider.loadGlobals() for example. I'm sure Pinaki will have a more information and a more robust example. One other thing I wasn't clear on is the type for the extension (TYPE_SPEC, TYPE_PRODUCT,etc). My first guess is that any extensions would use TYPE_PRODUCT (value 100), but I could see it being TYPE_PRODUCT_STORE as well. PersistenceProductDerivation is TYPE_SPEC (value 0) and the ProductDerivations are sorted in ascending order. If extensions are of TYPE_PRODUCT then the default Apache derivation will always be checked first. If we assume that when an extension is present it will be used more often than the default we might want to have any extensions' derivations called first. That's all I found. I didn't mean to steal Pinaki's thunder, just wanted to show that someone else has had some success. I've committed some additional work on this as I reviewed Pinaki's work. You should now be able to extend OpenJPA by creating a ProductDerivation like so:
public class MyProductDerivation extends AbstractProductDerivation { public int getType() { return TYPE_PRODUCT; } public boolean beforeConfigurationLoad(Configuration conf) { Value emf = conf.getValue(EntityManagerFactoryValue.KEY); if (emf == null) return false; emf.setDefault(MyEntityManagerFactoryClass.class.getName()); emf.setClassName(MyEntityManagerFactoryClass.getName()); return true; } } With your custom EntityManagerFactory class (which must extend EntityManagerFactoryImpl), you can also override newEntityManagerImpl(Broker) to subclass the EntityManager, and using your EntityManager subclass I believe you can act as a factory for custom QueryImpl, etc extensions. In addition to the PersistenceProvider.createEMF methods, the static toEntitymanagerFactory/toEntityManager methods in OpenJPAPersistence will also correctly return your custom subclasses. So you shouldn't need a custom ConfigurationProvider or a custom PersistenceProvider anymore. I'll be doing some more review of this later to be sure it actually works, barring someone else testing it and telling me it does. Thanks Abe, I was able to load a custom EMF with your changes. I might have more questions when I get a chance to experiment a little more.
With Pinaki's and Abe's assistance, this JIRA report was resolved. Thank you.
|
||||||||||||||||||||||||||||||||||||||||||||||||||
 |
|
Bug
Closed
Major
Allow OpenJPA to be extensible
> - You mention in several places about separating away the notion of
> specs and stores. In a general sense, I understand what these
> are. But, can you elaborate on how these types are used in the
> ConfigurationProvider and ProductDerivation interfaces?
What I meant was that the ProductDerivation interface has methods and
constants that imply knowledge of what a "spec" is and what a "store"
is: afterSepcificationSet(), TYPE_STORE, etc. These methods and
constants become meaningless when the interface is moved from kernel
to lib, because lib is code that is completely ignorant of what's
built on top of it. OpenJPA kernel understands that there might be
different spec facades built on it, and that there might be different
data stores plugged in, but lib code shouldn't be aware of those
concepts.
Actually, I wouldn't mind moving the
OpenJPAConfiguration.setSpecification() method to the base
Configuration interface and giving lib the notion of a spec, because
that's a sufficiently general idea. But lib certainly shouldn't know
anything about data stores -- that concept is very persistence-
specific. So I believe that at the very least, the TYPE_STORE stuff
has to be moved out of ProductDerivation and into something in the
kernel if ProductDerivation itself moves into lib. As I mentioned in
my original email, it might seem odd to maintain the strict
neutrality of lib code given that it's only used for OpenJPA, but we
do in fact build on that code with some non-persistence-aware Kodo
stuff, and as long as there is a separation of modules within
OpenJPA, I'd like to maintain the meaning of lib-as-neutral vs.
kernel-as-persistence-aware.
> Now that we need to return a ConfigurationProvider, would you
> expect that we just new up a ConfigurationProviderImpl and then
> just call across to the "load" methods on the implementation? Since we
> want to keep the ProductDerivations stateless, I'm not sure how else you were
> expecting to create a ConfigurationProvider to return on these "load" methods.
I would expect the ProductDerivation itself to do most of the load
work and to populate a new ConfigurationProvider with the parsed
state. The ProductDerivation itself would remain stateless, but
would contain the load logic. We can probably have just one
ConfigurationProviderImpl that will work for most derivations (i.e.
ConfigurationProviderImpl will probably not have to be JPA-specific
anymore, and can move into lib's conf package or somewhere where it
can be used by JDO, etc as well). I bet a slight rework of
MapConfigurationProvider would do the trick.
> - Now that ConfigurationProvider is bare, the
> ConfigurationTestConfigurationProvider doesn't have much
> function. I'll need to take a look to see if this is even required any longer.
Yeah, I'm sure tests will need updating.
> - Can you shed a bit more light on the Configurations class? It
> doesn't implement nor extend any interfaces or classes, but it
> seems to provide many of the same methods as ConfigurationProvider, but as
> statics. And, it's dependent on having a Provider. Can you explain the
> relationship of this class in the bigger picture and how you think it might be
> affected by these changes?
It's a utility class. Aside from the low-level utils it provides,
it's mainly there so that its static configuration methods can be
invoked without worrying about what services the system is configured
with. Configurations does the work of looking up the right
ConfigurationProvider using the services framework and applying it.
Otherwise, each component that used a ConfigurationProvider would
have to invoke the Services utilities itself to figure out which
ConfigurationProvider to use.
When ProductDerivation takes over, Configurations will change to use
ProductDerivations instead, and will subsume the functionality of
kernel's conf.ProductDerivations utility class.