After a few days of debugging and profiling I managed to find that the culprit was Xalan!
(See here as well http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6434840)
Thus I have updated the XSLT mediator to use in-memory ByteArray streams for transformations upto 8K messages (transform source or result size) and to use temporary files where this size is exceeded. This allows me to perform transformations that yield both XML and non-XML outputs of over 10MB in size without any problem (even larger transformations should work fine now). The fix also removed the previously hard coded System property "javax.xml.transform.TransformerFactory" which may now be left unspecified (i.e. use the JDK default) or set to what you desire (e.g. "org.apache.xalan.xsltc.trax.TransformerFactoryImpl") depending on the environment/JDK used. This fix is checked in with associated unit tests