I checked with Jeremy on the OpenJPA team yesterday and he agreed that the openjpa.RuntimeUnenhancedClasses property is ignored for the javaagent code and only used for dynamic runtime enhancement (subclassing), so setting that value in the EMF Map should not be required and is discouraged, as we tell OpenJPA users all the time NOT to use dynamic runtime enhancement.
For the javaagent, classLoadEnhancement=true is the default, which controls whether OpenJPA load-time class enhancement should be available in this JVM execution.
Starting in OpenJPA 2.0 - If a javaagent is not provided via the command line and OpenJPA is running on the Sun 1.6 SDK or IBM 1.6 JDK (SR8+), OpenJPA will attempt to dynamically load the runtime Enhancer. This support is provided as an ease of use feature and it is not recommended for use in a production system. Setting the property openjpa.DynamicEnhancementAgent to false will disable this function.
When then dynamic enhancer is loaded, the following informational message is logged:
[java] jpa.enhancement INFO [main] openjpa.Runtime - OpenJPA dynamically loaded the class enhancer. Any classes that were not enhanced at build time will be enhanced when they are loaded by the JVM.
So, is another module (like a deployer, annotation scanner or Tomcat) possibly loading the classes before the OpenJPA GBean is being loaded and having a chance to register the TransformerAgent?