[SENTRY-183] Sentry Policy Service goes into an unusable state when granting privileges. Subsequent access fail with a DataNucleusException: "Iteration request failed: SELECT ..."

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 1.4.0
• Assignee: Prasad Suresh Mujumdar
• Reporter: Lenni Kuff
• Created: 23/Apr/14 22:24
• Updated: 25/May/14 14:20
• Resolved: 25/May/14 14:20

Description

Sentry Policy Service goes into an unusable state after performing some sequence (TBD) of GRANT statements. Subsequent grants fail with a DataNucleusException: "Iteration request failed: SELECT ...". Restarting the service does not resolve the problem, I had to create a clean policy store DB.

Full stack trace:

java.lang.RuntimeException: Unknown error for request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:1, requestorUserName:lskuff, requestorGroupNames:[lskuff], roleName:all_functional_seq_snap, privilege:TSentryPrivilege(privilegeScope:TABLE, serverName:server1, dbName:functional_seq_snap, tableName:*, action:SELECT, createTime:1398290831637, grantorPrincipal:lskuff)), message: Iteration request failed : SELECT 'org.apache.sentry.provider.db.service.model.MSentryGroup' AS NUCLEUS_TYPE,"A1"."CREATE_TIME","A1"."GRANTOR_PRINCIPAL","A1"."GROUP_NAME","A1"."GROUP_ID" FROM "SENTRY_ROLE_GROUP_MAP" "A0" INNER JOIN "SENTRY_GROUP" "A1" ON "A0"."GROUP_ID" = "A1"."GROUP_ID" WHERE "A0"."ROLE_ID" = ?. Server Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed : SELECT 'org.apache.sentry.provider.db.service.model.MSentryGroup' AS NUCLEUS_TYPE,"A1"."CREATE_TIME","A1"."GRANTOR_PRINCIPAL","A1"."GROUP_NAME","A1"."GROUP_ID" FROM "SENTRY_ROLE_GROUP_MAP" "A0" INNER JOIN "SENTRY_GROUP" "A1" ON "A0"."GROUP_ID" = "A1"."GROUP_ID" WHERE "A0"."ROLE_ID" = ?
	at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
	at org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
	at org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:155)
	at org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:137)
	at org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilege(SentryStore.java:226)
	at org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:162)
	at org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:693)
	at org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:678)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:225)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
NestedThrowablesStackTrace:
org.postgresql.util.PSQLException: ERROR: current transaction is aborted, commands ignored until end of transaction block
	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2102)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1835)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:273)
	at com.jolbox.bonecp.PreparedStatementHandle.executeQuery(PreparedStatementHandle.java:172)
	at org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeQuery(ParamLoggingPreparedStatement.java:381)
	at org.datanucleus.store.rdbms.SQLController.executeStatementQuery(SQLController.java:504)
	at org.datanucleus.store.rdbms.scostore.JoinSetStore.iterator(JoinSetStore.java:932)
	at org.datanucleus.store.types.backed.Set.loadFromStore(Set.java:325)
	at org.datanucleus.store.types.backed.Set.load(Set.java:298)
	at org.datanucleus.state.JDOStateManager.loadUnloadedFields(JDOStateManager.java:2967)
	at org.datanucleus.state.JDOStateManager.runReachability(JDOStateManager.java:3657)
	at org.datanucleus.store.fieldmanager.ReachabilityFieldManager.processPersistable(ReachabilityFieldManager.java:69)
	at org.datanucleus.store.fieldmanager.ReachabilityFieldManager.storeObjectField(ReachabilityFieldManager.java:120)
	at org.datanucleus.state.JDOStateManager.providedObjectField(JDOStateManager.java:1272)
	at org.apache.sentry.provider.db.service.model.MSentryPrivilege.jdoProvideField(MSentryPrivilege.java)
	at org.apache.sentry.provider.db.service.model.MSentryPrivilege.jdoProvideFields(MSentryPrivilege.java)
	at org.datanucleus.state.JDOStateManager.provideFields(JDOStateManager.java:1349)
	at org.datanucleus.state.JDOStateManager.runReachability(JDOStateManager.java:3672)
	at org.datanucleus.ExecutionContextImpl.performReachabilityAtCommit(ExecutionContextImpl.java:4331)
	at org.datanucleus.ExecutionContextImpl.preCommit(ExecutionContextImpl.java:4130)
	at org.datanucleus.ExecutionContextImpl.transactionPreCommit(ExecutionContextImpl.java:615)
	at org.datanucleus.TransactionImpl.internalPreCommit(TransactionImpl.java:375)
	at org.datanucleus.TransactionImpl.commit(TransactionImpl.java:264)
	at org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:98)
	at org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:155)
	at org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:137)
	at org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilege(SentryStore.java:226)
	at org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:162)
	at org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:693)
	at org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:678)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:225)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

	at org.apache.sentry.service.thrift.Status.throwIfNotOk(Status.java:104)
	at org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClient.grantPrivilege(SentryPolicyServiceClient.java:203)
	at org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClient.grantTablePrivilege(SentryPolicyServiceClient.java:179)

Issue Links

• links to: Review #21894