Uploaded image for project: 'Sentry'
  1. Sentry
  2. SENTRY-1855

Improve scalability of permission delta updates

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.0.0, 2.1.0
    • Fix Version/s: None
    • Component/s: Sentry
    • Labels:
      None

      Description

      Looking at the latest stress runs, we noticed that some of the transactions could fail to commit to the database (with Duplicate key exception) after exhausting all the retries.
      This problem has become more evident if we have more number of clients connecting to Sentry to issue the permission updates. Was able to reproduce consistently with 15 clients doing 100 operations each.
      In the past we introduced exponential backoff (SENTRY-1821) so as part of test run increased the defaults to 750ms sleep and 20 retries. But even after this, the cluster still shows up the transaction failures. This change also increases the latency of every transaction in sentry.
      We need to revisit this and come up with a better way to solve this problem.

      2017-07-13 13:18:14,449 ERROR org.apache.sentry.provider.db.service.persistent.TransactionManager: The transaction has reached max retry number, Exception thrown when executing query
      javax.jdo.JDOException: Exception thrown when executing query
      	at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)
      	at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:252)
      	at org.apache.sentry.provider.db.service.persistent.SentryStore.getRole(SentryStore.java:294)
      	at org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilegeCore(SentryStore.java:645)
      	at org.apache.sentry.provider.db.service.persistent.SentryStore.access$500(SentryStore.java:101)
      	at org.apache.sentry.provider.db.service.persistent.SentryStore$11.execute(SentryStore.java:601)
      	at org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:159)
      	at org.apache.sentry.provider.db.service.persistent.TransactionManager.access$100(TransactionManager.java:63)
      	at org.apache.sentry.provider.db.service.persistent.TransactionManager$2.call(TransactionManager.java:213)
      --
      	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:971)
      	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
      	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
      	at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
      	at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
      	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2530)
      	at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1907)
      	at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2141)
      	at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1773)
      	... 33 more
      2017-07-13 13:18:14,450 ERROR org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: Unknown error for request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, requestorUserName:hive, roleName:2017_07_12_15_06_38_1_2_805, privileges:[TSentryPrivilege(privilegeScope:DATABASE, serverName:server1, dbName:2017_07_12_15_06_38_1_2, tableName:, URI:, action:*, createTime:1499904401222, grantOption:FALSE, columnName:)]), message: The transaction has reached max retry number, Exception thrown when executing query
      java.lang.Exception: The transaction has reached max retry number, Exception thrown when executing query
      	at org.apache.sentry.provider.db.service.persistent.TransactionManager$ExponentialBackoff.execute(TransactionManager.java:255)
      	at org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionBlocksWithRetry(TransactionManager.java:209)
      	at org.apache.sentry.provider.db.service.persistent.SentryStore.execute(SentryStore.java:3330)
      	at org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilege(SentryStore.java:593)
      	at org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:633)
      	at org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:256)
      	at org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:997)
      	at org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:982)
      	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
      	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
      

        Attachments

        1. SENTRY-1855.01-sentry-ha-redesign.patch
          10 kB
          Alex Kolbasov
        2. SENTRY-1855.002-sentry-ha-redesign.patch
          17 kB
          Na Li
        3. SENTRY-1855.003-master.patch
          20 kB
          Na Li
        4. SENTRY-1855.001.patch
          120 kB
          Na Li
        5. SENTRY-1855.002.patch
          126 kB
          Na Li

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                akolb Alex Kolbasov
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: