Critical Description
Under high load commons-dbcp (or commons-pool) exhibits thread safety issues and begins throwing various exceptions. I don't yet know the cause of the issue but it looks like a connection maybe handed out to multiple threads concurrently. Here's a few examples of the exceptions we are getting:
jvm 1 | Caused by: java.sql.SQLException: Attempted to use PooledConnection after closed() was called. jvm 1 | at org.apache.commons.dbcp.cpdsadapter.PooledConnectionImpl.assertOpen(PooledConnectionImpl.java:163) jvm 1 | at org.apache.commons.dbcp.cpdsadapter.PooledConnectionImpl.getConnection(PooledConnectionImpl.java:174) jvm 1 | at org.apache.commons.dbcp.datasources.InstanceKeyDataSource.getConnection(InstanceKeyDataSource.java:768) jvm 1 | at org.apache.commons.dbcp.datasources.InstanceKeyDataSource.getConnection(InstanceKeyDataSource.java:676) jvm 1 | at uk.co.webessence.kernel.database.DriverAdapterConnectionPool.acquireConnection(DriverAdapterConnectionPool.java:101) jvm 1 | ... 94 more
jvm 1 | Caused by: java.sql.SQLException: PooledConnection was reused, withoutits previous Connection being closed. jvm 1 | at org.apache.commons.dbcp.cpdsadapter.PooledConnectionImpl.getConnection(PooledConnectionImpl.java:179) jvm 1 | at org.apache.commons.dbcp.datasources.InstanceKeyDataSource.getConnection(InstanceKeyDataSource.java:768) jvm 1 | at org.apache.commons.dbcp.datasources.InstanceKeyDataSource.getConnection(InstanceKeyDataSource.java:676) jvm 1 | at uk.co.webessence.kernel.database.DriverAdapterConnectionPool.acquireConnection(DriverAdapterConnectionPool.java:101) jvm 1 | ... 77 more
jvm 1 | Caused by: java.sql.SQLException: Invalid state, the ResultSet object is closed. jvm 1 | at net.sourceforge.jtds.jdbc.JtdsResultSet.checkOpen(JtdsResultSet.java:299) jvm 1 | at net.sourceforge.jtds.jdbc.JtdsResultSet.getColumn(JtdsResultSet.java:273) jvm 1 | at net.sourceforge.jtds.jdbc.JtdsResultSet.getObject(JtdsResultSet.java:840) jvm 1 | at org.apache.commons.dbcp.DelegatingResultSet.getObject(DelegatingResultSet.java:325) jvm 1 | at uk.co.webessence.kernel.persistence.Preloader.getDataArray(Preloader.java:428) jvm 1 | at uk.co.webessence.kernel.persistence.Preloader.processSingleRow(Preloader.java:175) jvm 1 | at uk.co.webessence.kernel.persistence.PersistenceHandler.processRecordReload(PersistenceHandler.java:471) jvm 1 | at uk.co.webessence.kernel.persistence.PersistenceHandler$1.doLoad(PersistenceHandler.java:447) jvm 1 | ... 71 more
Message: TDS Protocol error: Invalid packet type 0x4 jvm 1 | Caused by: java.sql.SQLException: TDS Protocol error: Invalid packet type 0x4 jvm 1 | at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2314) jvm 1 | at net.sourceforge.jtds.jdbc.TdsCore.getNextRow(TdsCore.java:764) jvm 1 | at net.sourceforge.jtds.jdbc.JtdsResultSet.next(JtdsResultSet.java:593) jvm 1 | at org.apache.commons.dbcp.DelegatingResultSet.next(DelegatingResultSet.java:207) jvm 1 | at uk.co.webessence.kernel.persistence.Preloader.loadData(Preloader.java:142) jvm 1 | at uk.co.webessence.kernel.persistence.PersistenceHandler$3.doLoad(PersistenceHandler.java:592) jvm 1 | ... 111 more jvm 1 | Caused by: net.sourceforge.jtds.jdbc.ProtocolException: Invalid packet type 0x4 jvm 1 | at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2301) jvm 1 | ... 116 more