[CASSANDRA-15850] Delay between Gossip settle and CQL port opening during the startup

• Type: Improvement
• Status: Open
• Priority: Normal
• Resolution: Unresolved
• Fix Version/s: 5.x
• Component/s: Local/Startup and Shutdown
• Change Category: Performance
• Complexity: Normal
• Platform: All
• Assignee: Unassigned
• Reporter: Jai Bheemsen Rao Dhanwada
• Created: 04/Jun/20 01:46
• Updated: 07/Mar/23 10:54

Description

Hello,

When I am bootstrapping/restarting a Cassandra Node, there is a delay between gossip settle and CQL port opening. Can someone please explain me where this delay is configured and can this be changed? I don't see any information in the logs

In my case if you see there is a ~3 minutes delay and this increases if I increase the #of tables and #of nodes and DC.

INFO  [main] 2020-05-31 23:51:07,554 Gossiper.java:1692 - Waiting for gossip to settle...
INFO  [main] 2020-05-31 23:51:15,555 Gossiper.java:1723 - No gossip backlog; proceeding
INFO  [main] 2020-05-31 23:54:06,867 NativeTransportService.java:70 - Netty using native Epoll event loop
INFO  [main] 2020-05-31 23:54:06,913 Server.java:155 - Using Netty Version: [netty-buffer=netty-buffer-4.0.44.Final.452812a, netty-codec=netty-codec-4.0.44.Final.452812a, netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a, netty-codec-http=netty-codec-http-4.0.44.Final.452812a, netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a, netty-common=netty-common-4.0.44.Final.452812a, netty-handler=netty-handler-4.0.44.Final.452812a, netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb, netty-transport=netty-transport-4.0.44.Final.452812a, netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a, netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a, netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a, netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a]
INFO  [main] 2020-05-31 23:54:06,913 Server.java:156 - Starting listening for CQL clients on /x.x.x.x:9042 (encrypted)...

Also during this 3-10 minutes delay, I see

nodetool compactionstats

command is hung and never respond, until the CQL port is up and running.

Can someone please help me understand the delay here?

Cassandra Version: 3.11.3

The issue can be easily reproducible with around 300 Tables and 100 nodes in a cluster.