I found a serious memory leak in the Xalan library. It is caused by the org.apache.xalan.transformer.TransformerIdentityImpl class.
I try to explain mechanism how the memory leak is caused:
Web application is deployed on SAP NetWeaver AS with Java 1.6 (1.6.0_95). The application uses Spring WS library but also another libraries with dependency to Xalan library.
When the web service is invoked, then the TransformerIdentityImpl class is used. This class is used because it's extends javax.xml.transform.Transformer and it is created by Java core method - javax.xml.transform.TransformerFactory.newInstance().
A class com.sun.xml.internal.messaging.saaj.soap.EnvelopeFactory is used for handling web services. This object also contains cache (ParserPool) with SAXParser objects (com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl). As key for this cache is used an application class loader.
EnvelopeFactory object and SAXParserImpl objects are loaded by a system class loader, but TransformerIdentityImpl class is loaded by an application class loader.
During handling of web service a method org.apache.xalan.transformer.TransformerIdentityImpl.transform(Source, Result) is invoked. This method uses SAXParser as a reader. Problem is that this method register self for handling properties (eg: "http://xml.org/sax/handlers/DeclHandler") but after processing method doesn't unregister self from SAXParser. It means that objects loaded by the system class loaded has dependencies on objects loaded by the application class loader.
Objects are still loaded after application is undeployed because cached SAXParser references TransformerIdentityImpl.
I prepared very simple patch to fix this problem. In the attachment is also picture which demonstrates situation.