Major Description
Deleting services seems to use a lot of cursors, which could result in exceptions like this:
[http-bio-6080-exec-11] ERROR org.apache.ranger.rest.ServiceREST (ServiceREST.java:771) - deleteService(22) failed javax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException Internal Exception: java.sql.SQLException: ORA-00604: error occurred at recursive SQL level 1 ORA-01000: maximum open cursors exceeded ORA-00604: error occurred at recursive SQL level 1 ORA-01000: maximum open cursors exceeded Error Code: 604 Call: delete from x_policy_item_group_perm where policy_item_id=99 Query: DataModifyQuery(sql="delete from x_policy_item_group_perm where policy_item_id=99") at org.eclipse.persistence.internal.jpa.QueryImpl.executeUpdate(QueryImpl.java:308) at org.apache.ranger.common.db.BaseDao.deletePolicyIDReference(BaseDao.java:261) at org.apache.ranger.biz.ServiceDBStore.deleteExistingPolicyItemsNative(ServiceDBStore.java:3337) at org.apache.ranger.biz.ServiceDBStore.deletePolicy(ServiceDBStore.java:2022) at org.apache.ranger.biz.ServiceDBStore.deleteService(ServiceDBStore.java:1676) at org.apache.ranger.rest.ServiceREST.deleteService(ServiceREST.java:767) at org.apache.ranger.rest.ServiceREST$$FastClassBySpringCGLIB$$92dab672.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:701) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:64) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:633) at org.apache.ranger.rest.ServiceREST$$EnhancerBySpringCGLIB$$8fa8bfff.deleteService(<generated>) at org.apache.ranger.rest.PublicAPIsv2.deleteServiceByName(PublicAPIsv2.java:276) at org.apache.ranger.rest.PublicAPIsv2$$FastClassBySpringCGLIB$$b2e69455.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
Or
WARN com.mchange.v2.c3p0.impl.NewPooledConnection (NewPooledConnection.java:486) - [c3p0] A PooledConnection that has already signalled a Connection error is still in use! 2018-04-22 05:38:13,592 [http-bio-6080-exec-18] WARN com.mchange.v2.c3p0.impl.NewPooledConnection (NewPooledConnection.java:487) - [c3p0] Another error has occurred [ java.sql.SQLException: ORA-01000: maximum open cursors exceeded ] which will not be reported to listeners! java.sql.SQLException: ORA-01000: maximum open cursors exceeded at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:450) at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:399) at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1059) at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:522) at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:257) at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:587) at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:225) at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:53) at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:774) at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:925) at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1111) at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:4798) at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:4845) at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1501) at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:76) at org.eclipse.persistence.internal.databaseaccess.DatabasePlatform.wasFailureCommunicationBased(DatabasePlatform.java:2849) at org.eclipse.persistence.platform.database.OraclePlatform.wasFailureCommunicationBased(OraclePlatform.java:1021) at org.eclipse.persistence.platform.server.ServerPlatformBase.wasFailureCommunicationBased(ServerPlatformBase.java:528) at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.processExceptionForCommError(DatabaseAccessor.java:1607) at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:898) at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeNoSelect(DatabaseAccessor.java:962) at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:631) at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:558) at org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:2002) at org.eclipse.persistence.sessions.server.ClientSession.executeCall(ClientSession.java:298) at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:242) at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:228) at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeNoSelectCall(DatasourceCallQueryMechanism.java:271) at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeNoSelect(DatasourceCallQueryMechanism.java:251) at org.eclipse.persistence.queries.DataModifyQuery.executeDatabaseQuery(DataModifyQuery.java:85) at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:899) at org.eclipse.persistence.queries.DatabaseQuery.executeInUnitOfWork(DatabaseQuery.java:798) at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2896) at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1804) at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1786) at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1751) at org.eclipse.persistence.internal.jpa.QueryImpl.executeUpdate(QueryImpl.java:298) at org.apache.ranger.common.db.BaseDao.deletePolicyIDReference(BaseDao.java:261) at org.apache.ranger.biz.ServiceDBStore.deleteExistingPolicyItemsNative(ServiceDBStore.java:3336) at org.apache.ranger.biz.ServiceDBStore.deletePolicy(ServiceDBStore.java:2022) at org.apache.ranger.biz.ServiceDBStore.deleteService(ServiceDBStore.java:1676) at org.apache.ranger.rest.ServiceREST.deleteService(ServiceREST.java:767) at org.apache.ranger.rest.ServiceREST$$FastClassBySpringCGLIB$$92dab672.invoke(<generated>)
deleteService(14) failed javax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException Internal Exception: java.sql.SQLException: ORA-01000: maximum open cursors exceeded Error Code: 1000 Call: delete from x_policy_item_datamask where policy_item_id=48 Query: DataModifyQuery(sql="delete from x_policy_item_datamask where policy_item_id=48") at org.eclipse.persistence.internal.jpa.QueryImpl.executeUpdate(QueryImpl.java:308) at org.apache.ranger.common.db.BaseDao.deletePolicyIDReference(BaseDao.java:261) at org.apache.ranger.biz.ServiceDBStore.deleteExistingPolicyItemsNative(ServiceDBStore.java:3336) at org.apache.ranger.biz.ServiceDBStore.deletePolicy(ServiceDBStore.java:2022) at org.apache.ranger.biz.ServiceDBStore.deleteService(ServiceDBStore.java:1676) at org.apache.ranger.rest.ServiceREST.deleteService(ServiceREST.java:767)
Re-trying in a couple of seconds usually solves the issue, however it doesn't seem a good solution. Oracle is configured having 300 concurrent cursor open, so it's not clear, why this pool could be exhausted so easily)