[CONNECTORS-499] Connection pool can be shut down during tests before cleanup is complete

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: ManifoldCF 0.6
• Fix Version/s: ManifoldCF 1.0
• Component/s: Framework core
• Assignee: Karl Wright
• Reporter: Karl Wright
• Created: 03/Aug/12 09:03
• Updated: 13/Aug/12 16:14
• Resolved: 13/Aug/12 16:13

Description

Long-running tests against PostgreSQL and MySQL do not seem to clean up properly. The symptom is hang during cleanup:

 [junit] "Reference Handler" daemon prio=10 tid=0x00007f907c074800 nid=0x1a28 in Object.wait() [0x00007f9081900000]
    [junit]    java.lang.Thread.State: WAITING (on object monitor)
    [junit] 	at java.lang.Object.wait(Native Method)
    [junit] 	at java.lang.Object.wait(Object.java:502)
    [junit] 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
    [junit] 	- locked <0x00000000e81501f8> (a java.lang.ref.Reference$Lock)
    [junit] 
    [junit] "main" prio=10 tid=0x00007f907c00a000 nid=0x1a23 in Object.wait() [0x00007f908553d000]
    [junit]    java.lang.Thread.State: WAITING (on object monitor)
    [junit] 	at java.lang.Object.wait(Native Method)
    [junit] 	at java.lang.Object.wait(Object.java:502)
    [junit] 	at org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:76)
    [junit] 	- locked <0x00000000e80df1c0> (a org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
    [junit] 	at org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
    [junit] 	at org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
    [junit] 	at org.apache.manifoldcf.core.database.Database.internalTransactionBegin(Database.java:250)
    [junit] 	at org.apache.manifoldcf.core.database.Database.synchronizeTransactions(Database.java:237)
    [junit] 	at org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1382)
    [junit] 	at org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
    [junit] 	at org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
    [junit] 	at org.apache.manifoldcf.core.database.DBInterfaceMySQL.performQuery(DBInterfaceMySQL.java:769)
    [junit] 	at org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.java:221)
    [junit] 	at org.apache.manifoldcf.crawler.jobs.Jobs.checkIfReference(Jobs.java:2117)
    [junit] 	at org.apache.manifoldcf.crawler.jobs.JobManager.checkIfReference(JobManager.java:586)
    [junit] 	at org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.delete(RepositoryConnectionManager.java:476)
    [junit] 	at org.apache.manifoldcf.crawler.tests.ConnectorBaseMySQL.localCleanUp(ConnectorBaseMySQL.java:214)
    [junit] 	at org.apache.manifoldcf.crawler.tests.ConnectorBaseMySQL.cleanUp(ConnectorBaseMySQL.java:182)
    [junit] 	at org.apache.manifoldcf.crawler.tests.BaseITMySQL.cleanUp(BaseITMySQL.java:208)
    [junit] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    [junit] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    [junit] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    [junit] 	at java.lang.reflect.Method.invoke(Method.java:616)
    [junit] 	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
    [junit] 	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
    [junit] 	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
    [junit] 	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
    [junit] 	at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
    [junit] 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
    [junit] 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
    [junit] 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
    [junit] 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
    [junit] 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
    [junit] 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
    [junit] 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
    [junit] 	at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
    [junit] 	at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
    [junit] 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
    [junit] 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
    [junit] 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
    [junit] 
    [junit] "VM Thread" prio=10 tid=0x00007f907c06d800 nid=0x1a27 runnable 
    [junit] 
    [junit] "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f907c015000 nid=0x1a25 runnable 
    [junit] 
    [junit] "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f907c016800 nid=0x1a26 runnable 
    [junit] 
    [junit] "VM Periodic Task Thread" prio=10 tid=0x00007f907c09e800 nid=0x1a2e waiting on condition 
    [junit] 
    [junit] JNI global references: 1047

The reason for the hang is apparently that the connection pool has been shut down while the clean-up code is trying to de-register connectors. Obviously the last thing that should happen is the connection pool shut down.