[CASSANDRA-7135] Better checks and error messages for data and log dirs on startup - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Resolved
Priority: Normal
Resolution: Duplicate
Fix Version/s: 2.0.8
Component/s: None
Labels:
- lhf

Description

We used to have checks to make sure the data and logging dirs existed and had good permissions on startup. In 2.0, it seems that's not the case any more. If /var/lib/cassandra and /var/log/cassandra don't exist, you'll get a startup log like this:

~/cassandra $ bin/cassandra -f
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /var/log/cassandra/system.log (No such file or directory)
	at java.io.FileOutputStream.open(Native Method)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:142)
	at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
	at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207)
	at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
	at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
	at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
	at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
	at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:809)
	at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:735)
	at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:615)
	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:502)
	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:395)
	at org.apache.log4j.PropertyWatchdog.doOnChange(PropertyConfigurator.java:922)
	at org.apache.log4j.helpers.FileWatchdog.checkAndConfigure(FileWatchdog.java:89)
	at org.apache.log4j.helpers.FileWatchdog.<init>(FileWatchdog.java:58)
	at org.apache.log4j.PropertyWatchdog.<init>(PropertyConfigurator.java:914)
	at org.apache.log4j.PropertyConfigurator.configureAndWatch(PropertyConfigurator.java:461)
	at org.apache.cassandra.service.CassandraDaemon.initLog4j(CassandraDaemon.java:133)
	at org.apache.cassandra.service.CassandraDaemon.<clinit>(CassandraDaemon.java:81)
 INFO 18:10:23,635 Logging initialized
 INFO 18:10:23,670 Loading settings from file:/home/thobbs/cassandra/conf/cassandra.yaml
 INFO 18:10:24,061 Data files directories: [/var/lib/cassandra/data]
 INFO 18:10:24,062 Commit log directory: /var/lib/cassandra/commitlog
 INFO 18:10:24,062 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
 INFO 18:10:24,062 disk_failure_policy is stop
 INFO 18:10:24,062 commit_failure_policy is stop
 INFO 18:10:24,066 Global memtable threshold is enabled at 981MB
 INFO 18:10:24,186 Not using multi-threaded compaction
 INFO 18:10:24,393 JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0_40
 INFO 18:10:24,393 Heap size: 4116709376/4116709376
 INFO 18:10:24,393 Code Cache Non-heap memory: init = 2555904(2496K) used = 664064(648K) committed = 2555904(2496K) max = 50331648(49152K)
 INFO 18:10:24,393 Par Eden Space Heap memory: init = 671088640(655360K) used = 107378592(104861K) committed = 671088640(655360K) max = 671088640(655360K)
 INFO 18:10:24,393 Par Survivor Space Heap memory: init = 83886080(81920K) used = 0(0K) committed = 83886080(81920K) max = 83886080(81920K)
 INFO 18:10:24,393 CMS Old Gen Heap memory: init = 3361734656(3282944K) used = 0(0K) committed = 3361734656(3282944K) max = 3361734656(3282944K)
 INFO 18:10:24,394 CMS Perm Gen Non-heap memory: init = 21757952(21248K) used = 14180160(13847K) committed = 21757952(21248K) max = 85983232(83968K)
 INFO 18:10:24,394 Classpath: bin/../conf:bin/../build/classes/main:bin/../build/classes/thrift:bin/../lib/antlr-3.2.jar:bin/../lib/commons-cli-1.1.jar:bin/../lib/commons-codec-1.2.jar:bin/../lib/commons-lang3-3.1.jar:bin/../lib/compress-lzf-0.8.4.jar:bin/../lib/concurrentlinkedhashmap-lru-1.3.jar:bin/../lib/disruptor-3.0.1.jar:bin/../lib/guava-15.0.jar:bin/../lib/high-scale-lib-1.1.2.jar:bin/../lib/jackson-core-asl-1.9.2.jar:bin/../lib/jackson-mapper-asl-1.9.2.jar:bin/../lib/jamm-0.2.5.jar:bin/../lib/jbcrypt-0.3m.jar:bin/../lib/jline-1.0.jar:bin/../lib/json-simple-1.1.jar:bin/../lib/libthrift-0.9.1.jar:bin/../lib/log4j-1.2.16.jar:bin/../lib/lz4-1.2.0.jar:bin/../lib/metrics-core-2.2.0.jar:bin/../lib/netty-3.6.6.Final.jar:bin/../lib/reporter-config-2.1.0.jar:bin/../lib/servlet-api-2.5-20081211.jar:bin/../lib/slf4j-api-1.7.2.jar:bin/../lib/slf4j-log4j12-1.7.2.jar:bin/../lib/snakeyaml-1.11.jar:bin/../lib/snappy-java-1.0.5.jar:bin/../lib/snaptree-0.1.jar:bin/../lib/super-csv-2.1.0.jar:bin/../lib/thrift-server-0.3.3.jar:bin/../lib/jamm-0.2.5.jar
 INFO 18:10:24,395 JNA not found. Native methods will be disabled.
 INFO 18:10:24,410 Initializing key cache with capacity of 100 MBs.
 INFO 18:10:24,423 Scheduling key cache save to each 14400 seconds (going to save all keys).
 INFO 18:10:24,424 Initializing row cache with capacity of 0 MBs
 INFO 18:10:24,433 Scheduling row cache save to each 0 seconds (going to save all keys).
ERROR 18:10:24,461 Failed to create /var/lib/cassandra/data/system/schema_triggers directory
ERROR 18:10:24,468 Failed to create /var/lib/cassandra/data/system/schema_triggers directory
ERROR 18:10:24,469 Failed to create /var/lib/cassandra/data/system/schema_triggers directory
ERROR 18:10:24,469 Failed to create /var/lib/cassandra/data/system/schema_triggers directory
ERROR 18:10:24,470 Failed to create /var/lib/cassandra/data/system/schema_triggers directory
ERROR 18:10:24,470 Failed to create /var/lib/cassandra/data/system/compaction_history directory
ERROR 18:10:24,471 Failed to create /var/lib/cassandra/data/system/compaction_history directory
ERROR 18:10:24,471 Failed to create /var/lib/cassandra/data/system/compaction_history directory
ERROR 18:10:24,472 Failed to create /var/lib/cassandra/data/system/compaction_history directory
ERROR 18:10:24,472 Failed to create /var/lib/cassandra/data/system/compaction_history directory
ERROR 18:10:24,473 Failed to create /var/lib/cassandra/data/system/compaction_history directory
ERROR 18:10:24,474 Failed to create /var/lib/cassandra/data/system/compaction_history directory
ERROR 18:10:24,474 Failed to create /var/lib/cassandra/data/system/compaction_history directory
ERROR 18:10:24,475 Failed to create /var/lib/cassandra/data/system/batchlog directory
ERROR 18:10:24,475 Failed to create /var/lib/cassandra/data/system/batchlog directory
ERROR 18:10:24,476 Failed to create /var/lib/cassandra/data/system/batchlog directory
ERROR 18:10:24,476 Failed to create /var/lib/cassandra/data/system/batchlog directory
ERROR 18:10:24,477 Failed to create /var/lib/cassandra/data/system/batchlog directory
ERROR 18:10:24,477 Failed to create /var/lib/cassandra/data/system/sstable_activity directory
ERROR 18:10:24,479 Failed to create /var/lib/cassandra/data/system/sstable_activity directory
ERROR 18:10:24,480 Failed to create /var/lib/cassandra/data/system/sstable_activity directory
ERROR 18:10:24,480 Failed to create /var/lib/cassandra/data/system/sstable_activity directory
ERROR 18:10:24,481 Failed to create /var/lib/cassandra/data/system/sstable_activity directory
ERROR 18:10:24,481 Failed to create /var/lib/cassandra/data/system/sstable_activity directory
ERROR 18:10:24,482 Failed to create /var/lib/cassandra/data/system/peer_events directory
ERROR 18:10:24,482 Failed to create /var/lib/cassandra/data/system/peer_events directory
ERROR 18:10:24,483 Failed to create /var/lib/cassandra/data/system/peer_events directory
ERROR 18:10:24,484 Failed to create /var/lib/cassandra/data/system/compactions_in_progress directory
ERROR 18:10:24,484 Failed to create /var/lib/cassandra/data/system/compactions_in_progress directory
ERROR 18:10:24,485 Failed to create /var/lib/cassandra/data/system/compactions_in_progress directory
ERROR 18:10:24,485 Failed to create /var/lib/cassandra/data/system/compactions_in_progress directory
ERROR 18:10:24,486 Failed to create /var/lib/cassandra/data/system/compactions_in_progress directory
ERROR 18:10:24,486 Failed to create /var/lib/cassandra/data/system/hints directory
ERROR 18:10:24,487 Failed to create /var/lib/cassandra/data/system/hints directory
ERROR 18:10:24,488 Failed to create /var/lib/cassandra/data/system/hints directory
ERROR 18:10:24,488 Failed to create /var/lib/cassandra/data/system/hints directory
ERROR 18:10:24,489 Failed to create /var/lib/cassandra/data/system/hints directory
ERROR 18:10:24,489 Failed to create /var/lib/cassandra/data/system/schema_keyspaces directory
ERROR 18:10:24,490 Failed to create /var/lib/cassandra/data/system/schema_keyspaces directory
ERROR 18:10:24,490 Failed to create /var/lib/cassandra/data/system/schema_keyspaces directory
ERROR 18:10:24,491 Failed to create /var/lib/cassandra/data/system/schema_keyspaces directory
ERROR 18:10:24,491 Failed to create /var/lib/cassandra/data/system/schema_keyspaces directory
ERROR 18:10:24,492 Failed to create /var/lib/cassandra/data/system/range_xfers directory
ERROR 18:10:24,492 Failed to create /var/lib/cassandra/data/system/range_xfers directory
ERROR 18:10:24,493 Failed to create /var/lib/cassandra/data/system/range_xfers directory
ERROR 18:10:24,494 Failed to create /var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,494 Failed to create /var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,495 Failed to create /var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,495 Failed to create /var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,496 Failed to create /var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,496 Failed to create /var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,497 Failed to create /var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,498 Failed to create /var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,498 Failed to create /var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,499 Failed to create /var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,499 Failed to create /var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,500 Failed to create /var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,500 Failed to create /var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,501 Failed to create /var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,502 Failed to create /var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,502 Failed to create /var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,503 Failed to create /var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,504 Failed to create /var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,504 Failed to create /var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,505 Failed to create /var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,506 Failed to create /var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,506 Failed to create /var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,507 Failed to create /var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,507 Failed to create /var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,508 Failed to create /var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,508 Failed to create /var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,509 Failed to create /var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,510 Failed to create /var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,510 Failed to create /var/lib/cassandra/data/system/schema_columnfamilies directory
ERROR 18:10:24,511 Failed to create /var/lib/cassandra/data/system/NodeIdInfo directory
ERROR 18:10:24,511 Failed to create /var/lib/cassandra/data/system/NodeIdInfo directory
ERROR 18:10:24,512 Failed to create /var/lib/cassandra/data/system/NodeIdInfo directory
ERROR 18:10:24,512 Failed to create /var/lib/cassandra/data/system/NodeIdInfo directory
ERROR 18:10:24,513 Failed to create /var/lib/cassandra/data/system/paxos directory
ERROR 18:10:24,513 Failed to create /var/lib/cassandra/data/system/paxos directory
ERROR 18:10:24,514 Failed to create /var/lib/cassandra/data/system/paxos directory
ERROR 18:10:24,514 Failed to create /var/lib/cassandra/data/system/paxos directory
ERROR 18:10:24,515 Failed to create /var/lib/cassandra/data/system/paxos directory
ERROR 18:10:24,515 Failed to create /var/lib/cassandra/data/system/paxos directory
ERROR 18:10:24,516 Failed to create /var/lib/cassandra/data/system/paxos directory
ERROR 18:10:24,516 Failed to create /var/lib/cassandra/data/system/paxos directory
ERROR 18:10:24,517 Failed to create /var/lib/cassandra/data/system/schema_columns directory
ERROR 18:10:24,517 Failed to create /var/lib/cassandra/data/system/schema_columns directory
ERROR 18:10:24,518 Failed to create /var/lib/cassandra/data/system/schema_columns directory
ERROR 18:10:24,519 Failed to create /var/lib/cassandra/data/system/schema_columns directory
ERROR 18:10:24,519 Failed to create /var/lib/cassandra/data/system/schema_columns directory
ERROR 18:10:24,520 Failed to create /var/lib/cassandra/data/system/schema_columns directory
ERROR 18:10:24,520 Failed to create /var/lib/cassandra/data/system/schema_columns directory
ERROR 18:10:24,521 Failed to create /var/lib/cassandra/data/system/schema_columns directory
ERROR 18:10:24,521 Failed to create /var/lib/cassandra/data/system/schema_columns directory
ERROR 18:10:24,521 Failed to create /var/lib/cassandra/data/system/schema_columns directory
ERROR 18:10:24,522 Failed to create /var/lib/cassandra/data/system/IndexInfo directory
ERROR 18:10:24,522 Failed to create /var/lib/cassandra/data/system/IndexInfo directory
ERROR 18:10:24,522 Failed to create /var/lib/cassandra/data/system/IndexInfo directory
ERROR 18:10:24,523 Failed to create /var/lib/cassandra/data/system/IndexInfo directory
ERROR 18:10:24,523 Failed to create /var/lib/cassandra/data/system/peers directory
ERROR 18:10:24,523 Failed to create /var/lib/cassandra/data/system/peers directory
ERROR 18:10:24,524 Failed to create /var/lib/cassandra/data/system/peers directory
ERROR 18:10:24,524 Failed to create /var/lib/cassandra/data/system/peers directory
ERROR 18:10:24,524 Failed to create /var/lib/cassandra/data/system/peers directory
ERROR 18:10:24,525 Failed to create /var/lib/cassandra/data/system/peers directory
ERROR 18:10:24,525 Failed to create /var/lib/cassandra/data/system/peers directory
ERROR 18:10:24,526 Failed to create /var/lib/cassandra/data/system/peers directory
ERROR 18:10:24,526 Failed to create /var/lib/cassandra/data/system/peers directory
ERROR 18:10:24,526 Failed to create /var/lib/cassandra/data/system/peers directory
ERROR 18:10:24,527 Failed to create /var/lib/cassandra/data/system/local directory
ERROR 18:10:24,527 Failed to create /var/lib/cassandra/data/system/local directory
ERROR 18:10:24,527 Failed to create /var/lib/cassandra/data/system/local directory
ERROR 18:10:24,528 Failed to create /var/lib/cassandra/data/system/local directory
ERROR 18:10:24,528 Failed to create /var/lib/cassandra/data/system/local directory
ERROR 18:10:24,528 Failed to create /var/lib/cassandra/data/system/local directory
ERROR 18:10:24,529 Failed to create /var/lib/cassandra/data/system/local directory
ERROR 18:10:24,529 Failed to create /var/lib/cassandra/data/system/local directory
ERROR 18:10:24,530 Failed to create /var/lib/cassandra/data/system/local directory
ERROR 18:10:24,530 Failed to create /var/lib/cassandra/data/system/local directory
ERROR 18:10:24,530 Failed to create /var/lib/cassandra/data/system/local directory
ERROR 18:10:24,531 Failed to create /var/lib/cassandra/data/system/local directory
ERROR 18:10:24,531 Failed to create /var/lib/cassandra/data/system/local directory
ERROR 18:10:24,531 Failed to create /var/lib/cassandra/data/system/local directory
ERROR 18:10:24,532 Failed to create /var/lib/cassandra/data/system/local directory
ERROR 18:10:24,532 Failed to create /var/lib/cassandra/data/system/local directory
ERROR 18:10:24,535 Fatal error: java.io.IOException: Failed to mkdirs /var/lib/cassandra/data
Failed to mkdirs /var/lib/cassandra/data; unable to start server

To summarize, we need to:

Make sure the directories exist
- If not, suggest creating them or editing cassandra.yaml
Make sure the directories have good permissions
- If not, say what permissions are needed and link to instructions for fixing them

Attachments

Issue Links

duplicates

CASSANDRA-5818 Duplicated error messages on directory creation error at startup

Resolved

Activity

People

Assignee:: Unassigned

Reporter:: Tom Hobbs

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: 01/May/14 23:14

Updated:: 16/Apr/19 09:31

Resolved:: 02/May/14 15:49