[SENTRY-1771] HDFS client concurrently requests full permission update multiple times

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.0
• Component/s: Sentry
• Labels: hfs
• Assignee: Alex Kolbasov
• Reporter: Na Li
• Created: 17/May/17 20:27
• Updated: 24/Jul/17 14:51
• Resolved: 21/Jun/17 06:30

Description

When running performance test, we found there are OOM, and the memory dump shows that there are 6 requests from HDFS to get full update for permission when OOM happens.

Server sends back full update if the sequence number from client is 1 at DBUpdateForwarder.getAllUpdatesFrom()

Example of call stack at the Sentry server is

  166,999K (17.4%) (org.datanucleus.ExecutionContextThreadedImpl)
     <-- Java Local (org.datanucleus.ExecutionContextThreadedImpl) [@c01196f0,@c07dbb10]
  99,019K (10.3%) (j.u.HashSet, j.u.HashMap)
     <-- org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds <-- Java Local (org.datanucleus.ExecutionContextThreadedImpl) [@c01196f0,@c07dbb10]
  85,196K (8.9%) (org.datanucleus.identity.OIDImpl)
     <--  {j.u.HashSet} <-- org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds <-- Java Local (org.datanucleus.ExecutionContextThreadedImpl) [@c01196f0,@c07dbb10]
  64,606K (6.7%)
     <-- org.datanucleus.identity.OIDImpl.toString <--  {j.u.HashSet} <-- org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds <-- Java Local (org.datanucleus.ExecutionContextThreadedImpl) [@c01196f0,@c07dbb10]
  GC root stack trace:
    org.datanucleus.store.rdbms.scostore.JoinSetStore.iterator(JoinSetStore.java:914)
    org.datanucleus.store.types.backed.Set.loadFromStore(Set.java:323)
    org.datanucleus.store.types.backed.Set.initialise(Set.java:272)
    org.datanucleus.store.types.SCOUtils.createSCOWrapper(SCOUtils.java:256)
    org.datanucleus.store.types.SCOUtils.newSCOInstance(SCOUtils.java:142)
    org.datanucleus.store.rdbms.mapping.java.AbstractContainerMapping.replaceFieldWithWrapper(AbstractContainerMapping.java:399)
    org.datanucleus.store.rdbms.mapping.java.AbstractContainerMapping.postFetch(AbstractContainerMapping.java:417)
    org.datanucleus.store.rdbms.request.FetchRequest.execute(FetchRequest.java:420)
    org.datanucleus.store.rdbms.RDBMSPersistenceHandler.fetchObject(RDBMSPersistenceHandler.java:324)
    org.datanucleus.state.AbstractStateManager.loadFieldsFromDatastore(AbstractStateManager.java:1122)
    org.datanucleus.state.JDOStateManager.loadUnloadedFieldsInFetchPlan(JDOStateManager.java:3000)
    org.datanucleus.state.AbstractStateManager.loadFieldsInFetchPlan(AbstractStateManager.java:1064)
    org.datanucleus.ExecutionContextImpl.performDetachAllOnTxnEndPreparation(ExecutionContextImpl.java:4574)
    org.datanucleus.ExecutionContextImpl.preCommit(ExecutionContextImpl.java:4259)
    org.datanucleus.ExecutionContextImpl.transactionPreCommit(ExecutionContextImpl.java:654)
    org.datanucleus.TransactionImpl.internalPreCommit(TransactionImpl.java:379)
    org.datanucleus.TransactionImpl.commit(TransactionImpl.java:268)
    org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:98)
    org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:113)
    org.apache.sentry.provider.db.service.persistent.SentryStore.retrieveFullPermssionsImage(SentryStore.java:2225)
    org.apache.sentry.hdfs.PermImageRetriever.retrieveFullImage(PermImageRetriever.java:56)
    org.apache.sentry.hdfs.PermImageRetriever.retrieveFullImage(PermImageRetriever.java:40)
    org.apache.sentry.hdfs.DBUpdateForwarder.getAllUpdatesFrom(DBUpdateForwarder.java:85)
    org.apache.sentry.hdfs.SentryPlugin.getAllPermsUpdatesFrom(SentryPlugin.java:178)
    org.apache.sentry.hdfs.SentryHDFSServiceProcessor.get_all_authz_updates_from(SentryHDFSServiceProcessor.java:48)
    org.apache.sentry.hdfs.service.thrift.SentryHDFSService$Processor$get_all_authz_updates_from.getResult(SentryHDFSService.java:403)
    org.apache.sentry.hdfs.service.thrift.SentryHDFSService$Processor$get_all_authz_updates_from.getResult(SentryHDFSService.java:388)
    org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
    org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
    org.apache.sentry.hdfs.SentryHDFSServiceProcessorFactory$ProcessorWrapper.process(SentryHDFSServiceProcessorFactory.java:47)
    org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
    org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    java.lang.Thread.run(Thread.java:745)

Issue Links

• Parent Feature: SENTRY-872 Uber jira for HMS HA + Sentry HA redesign
• relates to: SENTRY-1669 HMSFollower should read current processed notification ID from database every time it runs
• relates to: SENTRY-1613 Add propagating logic for Perm/Path updates in Sentry service
• relates to: SENTRY-1773 Delta change cleaner should leave way more then a single entry intact