[CASSANDRA-10612] Protocol v3 upgrade tests on 2.1->3.0 path fail when compaction is interrupted

• Type: Sub-task
• Status: Resolved
• Priority: Normal
• Resolution: Cannot Reproduce
• Fix Version/s: 3.0.5
• Labels: dtest
• Assignee: Russ Hatch
• Reporter: Jim Witschey

Authors

Russ Hatch

• Created: 28/Oct/15 14:55
• Updated: 01/Aug/21 12:39
• Resolved: 15/Mar/16 21:40

Description

The following tests in the upgrade_through_versions dtest suite fail:

• upgrade_through_versions_test.py:TestRandomPartitionerUpgrade.rolling_upgrade_test
• upgrade_through_versions_test.py:TestRandomPartitionerUpgrade.rolling_upgrade_with_internode_ssl_test
• upgrade_through_versions_test.py:TestUpgradeThroughVersions.rolling_upgrade_with_internode_ssl_test
• upgrade_through_versions_test.py:TestUpgradeThroughVersions.rolling_upgrade_test
• upgrade_through_versions_test.py:TestUpgrade_from_2_1_latest_tag_to_cassandra_3_0_HEAD.rolling_upgrade_test
• upgrade_through_versions_test.py:TestUpgrade_from_2_1_latest_tag_to_cassandra_3_0_HEAD.rolling_upgrade_with_internode_ssl_test
• upgrade_through_versions_test.py:TestUpgrade_from_cassandra_2_1_HEAD_to_cassandra_3_0_latest_tag.rolling_upgrade_test
• upgrade_through_versions_test.py:TestUpgrade_from_cassandra_2_1_HEAD_to_cassandra_3_0_latest_tag.rolling_upgrade_with_internode_ssl_test
• upgrade_through_versions_test.py:TestUpgrade_from_cassandra_2_1_HEAD_to_cassandra_3_0_HEAD.rolling_upgrade_test

See this report:

http://cassci.datastax.com/view/Upgrades/job/cassandra_upgrade_2.1_to_3.0_proto_v3/10/testReport/

They fail with the following error:

A subprocess has terminated early. Subprocess statuses: Process-41 (is_alive: True), Process-42 (is_alive: False), Process-43 (is_alive: True), Process-44 (is_alive: False), attempting to terminate remaining subprocesses now.

and with logs that look like this:

Unexpected error in node1 node log: ['ERROR [SecondaryIndexManagement:1] 2015-10-27 00:06:52,335 CassandraDaemon.java:195 - Exception in thread Thread[SecondaryIndexManagement:1,5,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.cassandra.db.compaction.CompactionInterruptedException: Compaction interrupted: Secondary index build@41202370-7c3e-11e5-9331-6bb6e58f8b1b(upgrade, cf, 578160/1663620)bytes
at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:368) ~[main/:na]
at org.apache.cassandra.index.internal.CassandraIndex.buildBlocking(CassandraIndex.java:688) ~[main/:na]
at org.apache.cassandra.index.internal.CassandraIndex.lambda$getBuildIndexTask$206(CassandraIndex.java:658) ~[main/:na]
at org.apache.cassandra.index.internal.CassandraIndex$$Lambda$151/1841229245.call(Unknown Source) ~[na:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_51]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_51]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_51]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] Caused by: java.util.concurrent.ExecutionException: org.apache.cassandra.db.compaction.CompactionInterruptedException: Compaction interrupted: Secondary index build@41202370-7c3e-11e5-9331-6bb6e58f8b1b(upgrade, cf, 578160/

1663620)bytes
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_51]
at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_51]
at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:364) ~[main/:na]
... 7 common frames omitted Caused by: org.apache.cassandra.db.compaction.CompactionInterruptedException: Compaction interrupted: Secondary index build@41202370-7c3e-11e5-9331-6bb6e58f8b1b(upgrade, cf, 578160/1663620)bytes
at org.apache.cassandra.index.SecondaryIndexBuilder.build(SecondaryIndexBuilder.java:67) ~[main/:na]
at org.apache.cassandra.db.compaction.CompactionManager$11.run(CompactionManager.java:1269) ~[main/:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_51]
... 4 common frames omitted', 'ERROR [HintsDispatcher:2] 2015-10-27 00:08:48,520 CassandraDaemon.java:195 - Exception in thread Thread[HintsDispatcher:2,1,main]', 'ERROR [HintsDispatcher:2] 2015-10-27 00:11:58,336 CassandraDaemon.java:195 - Exception in thread Thread[HintsDispatcher:2,1,main]']