[CASSANDRA-16008] 2.2.17 fails to start up with ExceptionInInitializerError

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 2.2.19
• Component/s: Local/Config
• Bug Category: Correctness
• Severity: Normal
• Complexity: Low Hanging Fruit
• Discovered By: User Report
• Platform: All
• Since Version: 2.2.18

Source Control Link

https://github.com/apache/cassandra/commit/ed8005552376188abb81894fb8c9c611ebd6379a

Test and Documentation Plan

Applied CASSANDRA-7066 fix in C* 3.0/3.11 to `DatabaseDescriptor.java` broken by commit `257fb03`.

• Assignee: Erick Ramirez (deprecated)
• Reporter: Tianon Gravi

Authors

Erick Ramirez (deprecated)

Reviewers

Brandon Williams

• Created: 03/Aug/20 22:06
• Updated: 04/Mar/22 07:33
• Resolved: 23/Sep/20 15:08

Description

After the upgrade to 2.2.17, Cassandra fails to start with the following error:

INFO  20:28:57 JVM Arguments: [-Dcom.sun.management.jmxremote.port=7199, -Dcom.sun.management.jmxremote.ssl=false, -Dcom.sun.management.jmxremote.authenticate=false, -ea, -javaagent:/opt/cassandra/lib/jamm-0.3.0.jar, -XX:+CMSClassUnloadingEnabled, -XX:+UseThreadPriorities, -XX:ThreadPriorityPolicy=42, -Xms128m, -Xmx128m, -Xmn32m, -XX:+HeapDumpOnOutOfMemoryError, -Xss256k, -XX:StringTableSize=1000003, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, -XX:+CMSParallelRemarkEnabled, -XX:SurvivorRatio=8, -XX:MaxTenuringThreshold=1, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+UseTLAB, -XX:+PerfDisableSharedMem, -XX:CompileCommandFile=/etc/cassandra/hotspot_compiler, -XX:CMSWaitDuration=10000, -XX:+CMSParallelInitialMarkEnabled, -XX:+CMSEdenChunksRecordAlways, -XX:CMSWaitDuration=10000, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintHeapAtGC, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -XX:+PrintPromotionFailure, -Xloggc:/opt/cassandra/logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=10, -XX:GCLogFileSize=10M, -Djava.net.preferIPv4Stack=true, -Dcassandra.jmx.local.port=7199, -XX:+DisableExplicitGC, -Djava.library.path=/opt/cassandra/lib/sigar-bin, -Dcassandra.libjemalloc=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1, -XX:OnOutOfMemoryError=kill -9 %p, -Dlogback.configurationFile=logback.xml, -Dcassandra.logdir=/opt/cassandra/logs, -Dcassandra.storagedir=/opt/cassandra/data, -Dcassandra-foreground=yes]
WARN  20:28:57 Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
INFO  20:28:57 jemalloc seems to be preloaded from /usr/lib/x86_64-linux-gnu/libjemalloc.so.1
INFO  20:28:57 JMX is enabled to receive remote connections on port: 7199
WARN  20:28:57 OpenJDK is not recommended. Please upgrade to the newest Oracle Java release
INFO  20:28:57 Initializing SIGAR library
INFO  20:28:57 Checked OS settings and found them configured for optimal performance.
WARN  20:28:57 Directory /opt/cassandra/data/commitlog doesn't exist
WARN  20:28:57 Directory /opt/cassandra/data/saved_caches doesn't exist
Exception (java.lang.ExceptionInInitializerError) encountered during startup: null
java.lang.ExceptionInInitializerError
	at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:709)
	at org.apache.cassandra.service.StartupChecks$9.execute(StartupChecks.java:351)
	at org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:109)
	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:188)
	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:607)
	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:717)
Caused by: java.lang.IllegalArgumentException: Bad configuration; unable to start server: At least one DataFileDirectory must be specified
	at org.apache.cassandra.config.DatabaseDescriptor.createAllDirectories(DatabaseDescriptor.java:846)
	at org.apache.cassandra.db.Keyspace.<clinit>(Keyspace.java:66)
	... 6 more
ERROR 20:28:58 Exception encountered during startup
java.lang.ExceptionInInitializerError: null
	at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:709) ~[apache-cassandra-2.2.17.jar:2.2.17]
	at org.apache.cassandra.service.StartupChecks$9.execute(StartupChecks.java:351) ~[apache-cassandra-2.2.17.jar:2.2.17]
	at org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:109) ~[apache-cassandra-2.2.17.jar:2.2.17]
	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:188) [apache-cassandra-2.2.17.jar:2.2.17]
	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:607) [apache-cassandra-2.2.17.jar:2.2.17]
	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:717) [apache-cassandra-2.2.17.jar:2.2.17]
Caused by: java.lang.IllegalArgumentException: Bad configuration; unable to start server: At least one DataFileDirectory must be specified
	at org.apache.cassandra.config.DatabaseDescriptor.createAllDirectories(DatabaseDescriptor.java:846) ~[apache-cassandra-2.2.17.jar:2.2.17]
	at org.apache.cassandra.db.Keyspace.<clinit>(Keyspace.java:66) ~[apache-cassandra-2.2.17.jar:2.2.17]
	... 6 common frames omitted
	at org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:109)
	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:188)
	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:607)
	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:717)
Caused by: java.lang.IllegalArgumentException: Bad configuration; unable to start server: At least one DataFileDirectory must be specified
	at org.apache.cassandra.config.DatabaseDescriptor.createAllDirectories(DatabaseDescriptor.java:846)
	at org.apache.cassandra.db.Keyspace.<clinit>(Keyspace.java:66)
	... 6 more

I've traced this down to what I believe is the issue in https://github.com/apache/cassandra/commit/257fb0377343cbfdb58327da17f31d4eaed940f5, specifically https://github.com/apache/cassandra/commit/257fb0377343cbfdb58327da17f31d4eaed940f5#r40944000 – the addition of an empty value for data_file_directories needs to be accompanied with a change to DatabaseDescriptor.java to handle an empty array the same way as the previous nil value was (and seed the value of cassandra.storagedir into that empty array), as was done in https://github.com/apache/cassandra/commit/b09e60f72bb2f37235d9e9190c25db36371b3c18#diff-b66584c9ce7b64019b5db5a531deeda1 (which I believe is the origin of this change).