[CASSANDRA-13059] dtest failure in upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_1_x_To_indev_3_x.rolling_upgrade_test

• Type: Improvement
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.13, 3.11.0
• Component/s: Test/dtest/python
• Labels: dtest test-failure
• Assignee: Benjamin Lerer
• Reporter: Sean McCarthy

Authors

Benjamin Lerer

• Created: 19/Dec/16 15:36
• Updated: 16/Apr/19 09:30
• Resolved: 17/Mar/17 11:17

Description

example failure:

http://cassci.datastax.com/job/cassandra-3.X_large_dtest/16/testReport/upgrade_tests.upgrade_through_versions_test/TestUpgrade_current_2_1_x_To_indev_3_x/rolling_upgrade_test/

Error Message

Subprocess ['nodetool', '-h', 'localhost', '-p', '7100', ['upgradesstables', '-a']] exited with non-zero status; exit status: 2; 
stderr: error: null
-- StackTrace --
java.lang.AssertionError
	at org.apache.cassandra.db.rows.Rows.collectStats(Rows.java:88)
	at org.apache.cassandra.io.sstable.format.big.BigTableWriter$StatsCollector.applyToRow(BigTableWriter.java:237)
	at org.apache.cassandra.db.transform.BaseRows.applyOne(BaseRows.java:120)
	at org.apache.cassandra.db.transform.BaseRows.add(BaseRows.java:110)
	at org.apache.cassandra.db.transform.UnfilteredRows.add(UnfilteredRows.java:44)
	at org.apache.cassandra.db.transform.Transformation.add(Transformation.java:174)
	at org.apache.cassandra.db.transform.Transformation.apply(Transformation.java:140)
	at org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:171)
	at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:135)
	at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65)
	at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:141)
	at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:199)
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
	at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
	at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
	at org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:420)
	at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:312)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$204(NamedThreadFactory.java:81)
	at org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$5/1992550266.run(Unknown Source)
	at java.lang.Thread.run(Thread.java:745)

Stacktrace

  File "/usr/lib/python2.7/unittest/case.py", line 329, in run
    testMethod()
  File "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py", line 279, in rolling_upgrade_test
    self.upgrade_scenario(rolling=True)
  File "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py", line 345, in upgrade_scenario
    self.upgrade_to_version(version_meta, partial=True, nodes=(node,))
  File "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py", line 446, in upgrade_to_version
    node.nodetool('upgradesstables -a')
  File "/home/automaton/venv/local/lib/python2.7/site-packages/ccmlib/node.py", line 783, in nodetool
    return handle_external_tool_process(p, ['nodetool', '-h', 'localhost', '-p', str(self.jmx_port), cmd.split()])
  File "/home/automaton/venv/local/lib/python2.7/site-packages/ccmlib/node.py", line 1993, in handle_external_tool_process
    raise ToolError(cmd_args, rc, out, err)

Related failures:
http://cassci.datastax.com/job/cassandra-3.X_large_dtest/16/testReport/upgrade_tests.upgrade_through_versions_test/TestUpgrade_current_2_2_x_To_indev_3_x/rolling_upgrade_with_internode_ssl_test/

Issue Links

• is related to: CASSANDRA-13320 upgradesstables fails after upgrading from 2.1.x to 3.0.11
• requires: CASSANDRA-13320 upgradesstables fails after upgrading from 2.1.x to 3.0.11