[CASSANDRA-11615] cassandra-stress blocks when connecting to a big cluster

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.7, 3.7
• Component/s: Legacy/Tools
• Severity: Normal
• Assignee: Andy Tolbert
• Reporter: Eduard Tudenhoefner

Authors

Andy Tolbert

Reviewers

T Jake Luciani

• Created: 20/Apr/16 06:55
• Updated: 16/Apr/19 09:30
• Resolved: 06/May/16 13:13

Description

I had a 100 node cluster and was running

cassandra-stress read n=100 no-warmup cl=LOCAL_QUORUM -rate 'threads=20' 'limit=1000/s'

Based on the thread dump it looks like it's been blocked at https://github.com/apache/cassandra/blob/cassandra-3.0/tools/stress/src/org/apache/cassandra/stress/util/JavaDriverClient.java#L96

"Thread-20" #245 prio=5 os_prio=0 tid=0x00007f3781822000 nid=0x46c4 waiting for monitor entry [0x00007f36cc788000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.cassandra.stress.util.JavaDriverClient.prepare(JavaDriverClient.java:96)
        - waiting to lock <0x00000005c003d920> (a java.util.concurrent.ConcurrentHashMap)
        at org.apache.cassandra.stress.operations.predefined.CqlOperation$JavaDriverWrapper.createPreparedStatement(CqlOperation.java:314)
        at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:77)
        at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:109)
        at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:261)
        at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:327)
"Thread-19" #244 prio=5 os_prio=0 tid=0x00007f3781820000 nid=0x46c3 waiting for monitor entry [0x00007f36cc889000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.cassandra.stress.util.JavaDriverClient.prepare(JavaDriverClient.java:96)
        - waiting to lock <0x00000005c003d920> (a java.util.concurrent.ConcurrentHashMap)
        at org.apache.cassandra.stress.operations.predefined.CqlOperation$JavaDriverWrapper.createPreparedStatement(CqlOperation.java:314)
        at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:77)
        at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:109)
        at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:261)
        at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:327)

I was trying the same with with a smaller cluster (50 nodes) and it was working fine.