[CASSANDRA-11469] dtest failure in upgrade_internal_auth_test.TestAuthUpgrade.upgrade_to_22_test

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Labels: dtest
• Severity: Normal
• Assignee: Sam Tunnicliffe
• Reporter: Philip Thompson

Authors

Sam Tunnicliffe

• Created: 31/Mar/16 16:21
• Updated: 16/Apr/19 09:30
• Resolved: 18/May/18 06:00

Description

Upgrading from 2.1 to 2.2, we are seeing failures in upgrade_internal_auth_test.py. I have seen a variety of stack traces, but the most common is:

java.lang.IllegalArgumentException: Unknown keyspace/cf pair (system_auth.credentials)
	at org.apache.cassandra.db.Keyspace.getColumnFamilyStore(Keyspace.java:169) ~[main/:na]
	at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1383) ~[main/:na]
	at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1277) ~[main/:na]
	at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:221) ~[main/:na]
	at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:176) ~[main/:na]
	at org.apache.cassandra.auth.PasswordAuthenticator.doAuthenticate(PasswordAuthenticator.java:143) ~[main/:na]
	at org.apache.cassandra.auth.PasswordAuthenticator.authenticate(PasswordAuthenticator.java:85) ~[main/:na]
	at org.apache.cassandra.auth.PasswordAuthenticator.access$100(PasswordAuthenticator.java:53) ~[main/:na]
	at org.apache.cassandra.auth.PasswordAuthenticator$PlainTextSaslAuthenticator.getAuthenticatedUser(PasswordAuthenticator.java:181) ~[main/:na]
	at org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:78) ~[main/:na]
	at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507) [main/:na]
	at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401) [main/:na]
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final]
	at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final]
	at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_51]
	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) [main/:na]
	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [main/:na]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]

This failure is flaky, but not uncommon. I ran it 300 times here, and saw about 100 failures on CI:

http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/53/testReport/

This is a recent failure, and given the type of failure, it seems like potentially a bug and not a test issue. beobal, you may be most interested in looking at this?

Issue Links

• is superceded by: CASSANDRA-12813 NPE in auth for bootstrapping node