[CASSANDRA-15034] cassandra-stress fails to retry user profile insert and query operations

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Invalid
• Component/s: Tool/stress
• Severity: Normal
• Assignee: Dinesh Joshi
• Reporter: Shlomi Livne

Authors

Dinesh Joshi

Reviewers

Dinesh Joshi, Dmitry Kropachev

• Created: 25/Feb/19 21:03
• Updated: 23/Apr/20 07:14
• Resolved: 23/Apr/20 04:57

Description

cassandra-stress that is run with a user profile against a cluster will fail to try operations when a node is killed.

To reproduce:

1. Create a 3 ndoe cluster with ccm

   ccm create cas-3 --vnodes -n 1 --version=3.11.3``

2. start the cluster

    ccm start

3. run

   tools/bin/cassandra-stress user profile=stress.yaml n=100000 ops(insert=1) no-warmup cl=QUORUM -node 127.0.0.2 -rate threads=10

   or run

   tools/bin/cassandra-stress user profile=stress.yaml n=100000 ops(simple=1) no-warmup cl=QUORUM -node 127.0.0.2 -rate threads=10

4. while the stress is progressing kill a node

   ccm node1 stop --not-gently

5. wait for cassandra-stress to end

for the insert case check

ccm node2 cqlsh -e " select count(*) FROM stresscql.tb ;"

we are missing rows

for the query case (simple) the following errors will be reported

java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.io.IOException: Operation x10 on key(s) [49a26f35297303469236]: Error executing: (NoSuchElementException)
at org.apache.cassandra.stress.Operation.error(Operation.java:127)java.util.NoSuchElementException
at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:105)com.datastax.driver.core.exceptions.TransportException: [/127.0.0.1:9042] Connection has been closed
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException

profile: stress.yamlstress.yaml