Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-8194

Reading from Auth table should not be in the request path

    XMLWordPrintableJSON

    Details

      Description

      We use PasswordAuthenticator and PasswordAuthorizer. The system_auth has a RF of 10 per DC over 2 DCs. The permissions_validity_in_ms is 5 minutes.

      We still have few thousand requests failing each day with the trace below. The reason for this is read cache request realizing that cached entry has expired and doing a blocking request to refresh cache.

      We should have cache refreshed periodically only in the back ground. The user request should simply look at the cache and not try to refresh it.

      com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
      at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258)
      at com.google.common.cache.LocalCache.get(LocalCache.java:3990)
      at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994)
      at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878)
      at org.apache.cassandra.service.ClientState.authorize(ClientState.java:292)
      at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:172)
      at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:165)
      at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:149)
      at org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:75)
      at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:102)
      at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:113)
      at org.apache.cassandra.thrift.CassandraServer.execute_cql3_query(CassandraServer.java:1735)
      at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4162)
      at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4150)
      at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
      at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
      at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:206)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
      at java.lang.Thread.run(Thread.java:722)
      Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
      at org.apache.cassandra.auth.Auth.selectUser(Auth.java:256)
      at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:84)
      at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50)
      at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:68)
      at org.apache.cassandra.service.ClientState$1.load(ClientState.java:278)
      at org.apache.cassandra.service.ClientState$1.load(ClientState.java:275)
      at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589)
      at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374)
      at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337)
      at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252)
      ... 19 more
      Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
      at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105)
      at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:943)
      at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:828)
      at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:140)
      at org.apache.cassandra.auth.Auth.selectUser(Auth.java:245)
      ... 28 more
      ERROR [Thrift:17232] 2014-10-24 05:06:51,004 CustomTThreadPoolServer.java (line 224) Error occurred during processing of message.
      com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
      at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258)
      at com.google.common.cache.LocalCache.get(LocalCache.java:3990)
      at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994)
      at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878)
      at org.apache.cassandra.service.ClientState.authorize(ClientState.java:292)
      at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:172)
      at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:165)
      at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:149)
      at org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:116)
      at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:102)
      at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:113)
      at org.apache.cassandra.thrift.CassandraServer.execute_cql3_query(CassandraServer.java:1735)
      at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4162)
      at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4150)
      at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
      at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
      at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:206)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
      at java.lang.Thread.run(Thread.java:722)
      Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
      at org.apache.cassandra.auth.Auth.selectUser(Auth.java:256)
      at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:84)
      at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50)
      at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:68)
      at org.apache.cassandra.service.ClientState$1.load(ClientState.java:278)
      at org.apache.cassandra.service.ClientState$1.load(ClientState.java:275)
      at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589)
      at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374)
      at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337)
      at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252)
      ... 19 more
      Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
      at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105)
      at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:943)
      at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:828)
      at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:140)
      at org.apache.cassandra.auth.Auth.selectUser(Auth.java:245)
      ... 28 more

        Attachments

        1. 8194.patch
          5 kB
          Vishy Kasar
        2. 8194-V2.patch
          4 kB
          Vishy Kasar
        3. CacheTest2.java
          2 kB
          Vishy Kasar
        4. 8194-V3.txt
          5 kB
          Sam Tunnicliffe
        5. 8194-V4.txt
          6 kB
          Sam Tunnicliffe
        6. 8194-V5.txt
          15 kB
          Sam Tunnicliffe

          Issue Links

            Activity

              People

              • Assignee:
                samt Sam Tunnicliffe
                Reporter:
                vkasar Vishy Kasar
                Authors:
                Sam Tunnicliffe
                Reviewers:
                Aleksey Yeschenko
              • Votes:
                1 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: