Details
-
Bug
-
Status: Resolved
-
Low
-
Resolution: Fixed
-
None
-
None
-
RHEL 6.5 (likely not hardware specific)
-
Low
Description
For example, when starting a newly installed Cassandra node for the first time, if the seed is down (and the current node is not a seed) a RuntimeException gets thrown by the Gossiper with "Unable to gossip with any seeds", and Cassandra subsequently shuts down.
The issue is with the StorageServiceShutdownHook thread which, if any exception is thrown as it's executing, will miss the remaining steps in the shutdown process.
Here's a log fragment.
INFO [MemoryMeter:1] 2014-01-29 09:07:04,126 Memtable.java (line 451) CFS(Keyspace='system', ColumnFamily='schema_columnfamilies') liveRatio is 2.8177450396801733 (just-counted was 2.5754099604780003). calculation took 459ms for 26900 cells
INFO [ScheduledTasks:1] 2014-01-29 09:07:04,168 GCInspector.java (line 116) GC for ParNew: 293 ms for 1 collections, 318930288 used; max is 6215958528
INFO [MemoryMeter:1] 2014-01-29 09:07:04,246 Memtable.java (line 451) CFS(Keyspace='system', ColumnFamily='schema_columns') liveRatio is 2.8940509915872097 (just-counted was 2.6458758011978563). calculation took 119ms for 23968 cells
INFO [MemoryMeter:1] 2014-01-29 09:07:07,005 Memtable.java (line 451) CFS(Keyspace='system', ColumnFamily='schema_columnfamilies') liveRatio is 2.6841006146942297 (just-counted was 2.550456189708286). calculation took 297ms for 54325 cells
INFO [ScheduledTasks:1] 2014-01-29 09:07:07,326 GCInspector.java (line 116) GC for ParNew: 289 ms for 1 collections, 291162792 used; max is 6215958528
INFO [MemoryMeter:1] 2014-01-29 09:07:07,607 Memtable.java (line 451) CFS(Keyspace='system', ColumnFamily='schema_columns') liveRatio is 2.770798679512837 (just-counted was 2.6475463674384643). calculation took 601ms for 47908 cells
INFO [main] 2014-01-29 09:07:09,914 StorageService.java (line 490) Cassandra version: 2.0.4-SNAPSHOT
INFO [main] 2014-01-29 09:07:09,914 StorageService.java (line 491) Thrift API version: 19.39.0
INFO [main] 2014-01-29 09:07:09,918 StorageService.java (line 492) CQL supported versions: 2.0.0,3.1.3 (default: 3.1.3)
INFO [main] 2014-01-29 09:07:09,933 StorageService.java (line 515) Loading persisted ring state
INFO [main] 2014-01-29 09:07:09,974 MessagingService.java (line 458) Starting Messaging Service on port 7000
ERROR [main] 2014-01-29 09:07:40,993 CassandraDaemon.java (line 473) Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any seeds
at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1160)
at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:426)
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:618)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:586)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:485)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:341)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:456)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:499)
ERROR [StorageServiceShutdownHook] 2014-01-29 09:07:41,150 CassandraDaemon.java (line 188) Exception in thread Thread[StorageServiceShutdownHook,5,main]
java.lang.NullPointerException
at org.apache.cassandra.service.StorageService.stopNativeTransport(StorageService.java:349)
at org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:364)
at org.apache.cassandra.service.StorageService.access$000(StorageService.java:97)
at org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:551)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.lang.Thread.run(Thread.java:744)