[CASSANDRA-13025] dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x.static_columns_with_distinct_test

• Type: Bug
• Status: Resolved
• Priority: Urgent
• Resolution: Fixed
• Fix Version/s: 3.0.11, 3.10
• Component/s: Test/dtest/python
• Labels: dtest test-failure
• Severity: Critical
• Assignee: Sylvain Lebresne Assign to me
• Reporter: Sean McCarthy

Authors

Sylvain Lebresne

Reviewers

Benjamin Lerer

• Created: 09/Dec/16 17:04
• Updated: 16/Apr/19 09:30
• Resolved: 27/Jan/17 08:33

Description

example failure:

http://cassci.datastax.com/job/cassandra-3.X_dtest_upgrade/28/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x/static_columns_with_distinct_test

Error Message

<Error from server: code=0000 [Server error] message="java.io.IOError: java.io.IOException: Corrupt empty row found in unfiltered partition">

Stacktrace

  File "/usr/lib/python2.7/unittest/case.py", line 329, in run
    testMethod()
  File "/home/automaton/cassandra-dtest/tools/decorators.py", line 46, in wrapped
    f(obj)
  File "/home/automaton/cassandra-dtest/upgrade_tests/cql_tests.py", line 4010, in static_columns_with_distinct_test
    rows = list(cursor.execute("SELECT DISTINCT k, s1 FROM test2"))
  File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line 1998, in execute
    return self.execute_async(query, parameters, trace, custom_payload, timeout, execution_profile, paging_state).result()
  File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line 3784, in result
    raise self._final_exception

Standard Output

http://git-wip-us.apache.org/repos/asf/cassandra.git git:7eac22dd41cb09e6d64fb5ac48b2cca3c8840cc8
Unexpected error in node2 log, error: 
ERROR [Native-Transport-Requests-2] 2016-12-08 03:20:04,861 Message.java:617 - Unexpected exception during request; channel = [id: 0xf4c13f2c, L:/127.0.0.2:9042 - R:/127.0.0.1:52112]
java.io.IOError: java.io.IOException: Corrupt empty row found in unfiltered partition
	at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:224) ~[apache-cassandra-3.9.jar:3.9]
	at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:212) ~[apache-cassandra-3.9.jar:3.9]
	at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.9.jar:3.9]
	at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:133) ~[apache-cassandra-3.9.jar:3.9]
	at org.apache.cassandra.cql3.statements.SelectStatement.processPartition(SelectStatement.java:779) ~[apache-cassandra-3.9.jar:3.9]
	at org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:741) ~[apache-cassandra-3.9.jar:3.9]
	at org.apache.cassandra.cql3.statements.SelectStatement.processResults(SelectStatement.java:408) ~[apache-cassandra-3.9.jar:3.9]
	at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:273) ~[apache-cassandra-3.9.jar:3.9]
	at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:232) ~[apache-cassandra-3.9.jar:3.9]
	at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:76) ~[apache-cassandra-3.9.jar:3.9]
	at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) ~[apache-cassandra-3.9.jar:3.9]
	at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:219) ~[apache-cassandra-3.9.jar:3.9]
	at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:204) ~[apache-cassandra-3.9.jar:3.9]
	at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:115) ~[apache-cassandra-3.9.jar:3.9]
	at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:513) [apache-cassandra-3.9.jar:3.9]
	at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:407) [apache-cassandra-3.9.jar:3.9]
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.39.Final.jar:4.0.39.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366) [netty-all-4.0.39.Final.jar:4.0.39.Final]
	at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.39.Final.jar:4.0.39.Final]
	at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357) [netty-all-4.0.39.Final.jar:4.0.39.Final]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_51]
	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) [apache-cassandra-3.9.jar:3.9]
	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [apache-cassandra-3.9.jar:3.9]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
Caused by: java.io.IOException: Corrupt empty row found in unfiltered partition
	at org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:430) ~[apache-cassandra-3.9.jar:3.9]
	at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:219) ~[apache-cassandra-3.9.jar:3.9]
	... 23 common frames omitted

Related failures: (~25)
http://cassci.datastax.com/job/cassandra-3.X_dtest_upgrade/28/testReport/

Issue Links

• is duplicated by: CASSANDRA-13026 dtest failure in upgrade_tests.paging_test.TestPagingDataNodes3RF3_Upgrade_current_3_x_To_indev_trunk.static_columns_paging_test