Uploaded image for project: 'Sentry'
  1. Sentry
  2. SENTRY-872 Uber jira for HMS HA + Sentry HA redesign
  3. SENTRY-1771

HDFS client concurrently requests full permission update multiple times

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.0.0
    • Component/s: Sentry
    • Labels:

      Description

      When running performance test, we found there are OOM, and the memory dump shows that there are 6 requests from HDFS to get full update for permission when OOM happens.

      Server sends back full update if the sequence number from client is 1 at DBUpdateForwarder.getAllUpdatesFrom()

      Example of call stack at the Sentry server is

        166,999K (17.4%) (org.datanucleus.ExecutionContextThreadedImpl)
           <-- Java Local (org.datanucleus.ExecutionContextThreadedImpl) [@c01196f0,@c07dbb10]
        99,019K (10.3%) (j.u.HashSet, j.u.HashMap)
           <-- org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds <-- Java Local (org.datanucleus.ExecutionContextThreadedImpl) [@c01196f0,@c07dbb10]
        85,196K (8.9%) (org.datanucleus.identity.OIDImpl)
           <--  {j.u.HashSet} <-- org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds <-- Java Local (org.datanucleus.ExecutionContextThreadedImpl) [@c01196f0,@c07dbb10]
        64,606K (6.7%)
           <-- org.datanucleus.identity.OIDImpl.toString <--  {j.u.HashSet} <-- org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds <-- Java Local (org.datanucleus.ExecutionContextThreadedImpl) [@c01196f0,@c07dbb10]
        GC root stack trace:
          org.datanucleus.store.rdbms.scostore.JoinSetStore.iterator(JoinSetStore.java:914)
          org.datanucleus.store.types.backed.Set.loadFromStore(Set.java:323)
          org.datanucleus.store.types.backed.Set.initialise(Set.java:272)
          org.datanucleus.store.types.SCOUtils.createSCOWrapper(SCOUtils.java:256)
          org.datanucleus.store.types.SCOUtils.newSCOInstance(SCOUtils.java:142)
          org.datanucleus.store.rdbms.mapping.java.AbstractContainerMapping.replaceFieldWithWrapper(AbstractContainerMapping.java:399)
          org.datanucleus.store.rdbms.mapping.java.AbstractContainerMapping.postFetch(AbstractContainerMapping.java:417)
          org.datanucleus.store.rdbms.request.FetchRequest.execute(FetchRequest.java:420)
          org.datanucleus.store.rdbms.RDBMSPersistenceHandler.fetchObject(RDBMSPersistenceHandler.java:324)
          org.datanucleus.state.AbstractStateManager.loadFieldsFromDatastore(AbstractStateManager.java:1122)
          org.datanucleus.state.JDOStateManager.loadUnloadedFieldsInFetchPlan(JDOStateManager.java:3000)
          org.datanucleus.state.AbstractStateManager.loadFieldsInFetchPlan(AbstractStateManager.java:1064)
          org.datanucleus.ExecutionContextImpl.performDetachAllOnTxnEndPreparation(ExecutionContextImpl.java:4574)
          org.datanucleus.ExecutionContextImpl.preCommit(ExecutionContextImpl.java:4259)
          org.datanucleus.ExecutionContextImpl.transactionPreCommit(ExecutionContextImpl.java:654)
          org.datanucleus.TransactionImpl.internalPreCommit(TransactionImpl.java:379)
          org.datanucleus.TransactionImpl.commit(TransactionImpl.java:268)
          org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:98)
          org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:113)
          org.apache.sentry.provider.db.service.persistent.SentryStore.retrieveFullPermssionsImage(SentryStore.java:2225)
          org.apache.sentry.hdfs.PermImageRetriever.retrieveFullImage(PermImageRetriever.java:56)
          org.apache.sentry.hdfs.PermImageRetriever.retrieveFullImage(PermImageRetriever.java:40)
          org.apache.sentry.hdfs.DBUpdateForwarder.getAllUpdatesFrom(DBUpdateForwarder.java:85)
          org.apache.sentry.hdfs.SentryPlugin.getAllPermsUpdatesFrom(SentryPlugin.java:178)
          org.apache.sentry.hdfs.SentryHDFSServiceProcessor.get_all_authz_updates_from(SentryHDFSServiceProcessor.java:48)
          org.apache.sentry.hdfs.service.thrift.SentryHDFSService$Processor$get_all_authz_updates_from.getResult(SentryHDFSService.java:403)
          org.apache.sentry.hdfs.service.thrift.SentryHDFSService$Processor$get_all_authz_updates_from.getResult(SentryHDFSService.java:388)
          org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
          org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
          org.apache.sentry.hdfs.SentryHDFSServiceProcessorFactory$ProcessorWrapper.process(SentryHDFSServiceProcessorFactory.java:47)
          org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
          org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
          java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          java.lang.Thread.run(Thread.java:745)
      
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                akolb Alexander Kolbasov
                Reporter:
                LinaAtAustin Na Li
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: