Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-11615

cassandra-stress blocks when connecting to a big cluster

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Fix Version/s: 3.0.7, 3.7
    • Component/s: Legacy/Tools
    • Labels:
      None
    • Severity:
      Normal

      Description

      I had a 100 node cluster and was running

      cassandra-stress read n=100 no-warmup cl=LOCAL_QUORUM -rate 'threads=20' 'limit=1000/s'
      

      Based on the thread dump it looks like it's been blocked at https://github.com/apache/cassandra/blob/cassandra-3.0/tools/stress/src/org/apache/cassandra/stress/util/JavaDriverClient.java#L96

      "Thread-20" #245 prio=5 os_prio=0 tid=0x00007f3781822000 nid=0x46c4 waiting for monitor entry [0x00007f36cc788000]
         java.lang.Thread.State: BLOCKED (on object monitor)
              at org.apache.cassandra.stress.util.JavaDriverClient.prepare(JavaDriverClient.java:96)
              - waiting to lock <0x00000005c003d920> (a java.util.concurrent.ConcurrentHashMap)
              at org.apache.cassandra.stress.operations.predefined.CqlOperation$JavaDriverWrapper.createPreparedStatement(CqlOperation.java:314)
              at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:77)
              at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:109)
              at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:261)
              at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:327)
      "Thread-19" #244 prio=5 os_prio=0 tid=0x00007f3781820000 nid=0x46c3 waiting for monitor entry [0x00007f36cc889000]
         java.lang.Thread.State: BLOCKED (on object monitor)
              at org.apache.cassandra.stress.util.JavaDriverClient.prepare(JavaDriverClient.java:96)
              - waiting to lock <0x00000005c003d920> (a java.util.concurrent.ConcurrentHashMap)
              at org.apache.cassandra.stress.operations.predefined.CqlOperation$JavaDriverWrapper.createPreparedStatement(CqlOperation.java:314)
              at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:77)
              at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:109)
              at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:261)
              at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:327)
      

      I was trying the same with with a smaller cluster (50 nodes) and it was working fine.

        Attachments

        1. 11615-3.0-2nd.patch
          6 kB
          Eduard Tudenhoefner
        2. 11615-3.0.patch
          1 kB
          Eduard Tudenhoefner

          Activity

            People

            • Assignee:
              andrew.tolbert Andy Tolbert
              Reporter:
              eduard.tudenhoefner Eduard Tudenhoefner
              Authors:
              Andy Tolbert
              Reviewers:
              T Jake Luciani
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: