Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-15850

Delay between Gossip settle and CQL port opening during the startup

    XMLWordPrintableJSON

    Details

    • Change Category:
      Performance
    • Complexity:
      Normal
    • Platform:
      All

      Description

      Hello,

      When I am bootstrapping/restarting a Cassandra Node, there is a delay between gossip settle and CQL port opening. Can someone please explain me where this delay is configured and can this be changed? I don't see any information in the logs

      In my case if you see there is a ~3 minutes delay and this increases if I increase the #of tables and #of nodes and DC.

      INFO  [main] 2020-05-31 23:51:07,554 Gossiper.java:1692 - Waiting for gossip to settle...
      INFO  [main] 2020-05-31 23:51:15,555 Gossiper.java:1723 - No gossip backlog; proceeding
      INFO  [main] 2020-05-31 23:54:06,867 NativeTransportService.java:70 - Netty using native Epoll event loop
      INFO  [main] 2020-05-31 23:54:06,913 Server.java:155 - Using Netty Version: [netty-buffer=netty-buffer-4.0.44.Final.452812a, netty-codec=netty-codec-4.0.44.Final.452812a, netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a, netty-codec-http=netty-codec-http-4.0.44.Final.452812a, netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a, netty-common=netty-common-4.0.44.Final.452812a, netty-handler=netty-handler-4.0.44.Final.452812a, netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb, netty-transport=netty-transport-4.0.44.Final.452812a, netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a, netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a, netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a, netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a]
      INFO  [main] 2020-05-31 23:54:06,913 Server.java:156 - Starting listening for CQL clients on /x.x.x.x:9042 (encrypted)...
      

      Also during this 3-10 minutes delay, I see

      nodetool compactionstats
      

      command is hung and never respond, until the CQL port is up and running.

      Can someone please help me understand the delay here?

      Cassandra Version: 3.11.3

      The issue can be easily reproducible with around 300 Tables and 100 nodes in a cluster.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jaid Jai Bheemsen Rao Dhanwada
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: