[CASSANDRA-2206] Startup fails due to cassandra trying to delete nonexisting file

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 0.7.3

Environment

linux

• Severity: Normal
• Assignee: Jonathan Ellis
• Reporter: Thibaut

Authors

Jonathan Ellis

Reviewers

Gary Dusbabek

• Created: 21/Feb/11 11:57
• Updated: 16/Apr/19 09:33
• Resolved: 24/Feb/11 15:14

Description

Hi,

On one of our nodes, startup fails due to cassandra trying to delete a nonexistant "Data" file (see below).

Why that file is missing is another mistery... The log file entries don't show any ERROR messages before cassandra restarted (for reasons I don't know) and this error occured.

Directory listing:

total 109M
rw-rr- 1 root root 51M 2011-02-21 05:25 table_task-f-1666-Data.db
rw-rr- 1 root root 243K 2011-02-21 05:25 table_task-f-1666-Filter.db
rw-rr- 1 root root 6.1M 2011-02-21 05:25 table_task-f-1666-Index.db
rw-rr- 1 root root 4.2K 2011-02-21 05:25 table_task-f-1666-Statistics.db
rw-rr- 1 root root 9.8M 2011-02-21 11:36 table_task-f-1703-Data.db
rw-rr- 1 root root 57K 2011-02-21 11:36 table_task-f-1703-Filter.db
rw-rr- 1 root root 1.3M 2011-02-21 11:36 table_task-f-1703-Index.db
rw-rr- 1 root root 4.2K 2011-02-21 11:36 table_task-f-1703-Statistics.db
rw-rr- 1 root root 292K 2011-02-21 11:42 table_task-f-1704-Data.db
rw-rr- 1 root root 1.7K 2011-02-21 11:42 table_task-f-1704-Filter.db
rw-rr- 1 root root 42K 2011-02-21 11:42 table_task-f-1704-Index.db
rw-rr- 1 root root 4.2K 2011-02-21 11:42 table_task-f-1704-Statistics.db
rw-rr- 1 root root 364K 2011-02-21 11:52 table_task-f-1705-Data.db
rw-rr- 1 root root 2.0K 2011-02-21 11:52 table_task-f-1705-Filter.db
rw-rr- 1 root root 50K 2011-02-21 11:52 table_task-f-1705-Index.db
rw-rr- 1 root root 4.2K 2011-02-21 11:52 table_task-f-1705-Statistics.db
rw-rr- 1 root root 535K 2011-02-21 12:10 table_task-f-1706-Data.db
rw-rr- 1 root root 2.8K 2011-02-21 12:10 table_task-f-1706-Filter.db
rw-rr- 1 root root 70K 2011-02-21 12:10 table_task-f-1706-Index.db
rw-rr- 1 root root 4.2K 2011-02-21 12:10 table_task-f-1706-Statistics.db
rw-rr- 1 root root 11M 2011-02-21 12:11 table_task-f-1707-Data.db
rw-rr- 1 root root 18M 2011-02-21 09:47 table_task_meta-f-417-Data.db
rw-rr- 1 root root 271K 2011-02-21 09:47 table_task_meta-f-417-Filter.db
rw-rr- 1 root root 6.7M 2011-02-21 09:47 table_task_meta-f-417-Index.db
rw-rr- 1 root root 4.2K 2011-02-21 09:47 table_task_meta-f-417-Statistics.db
rw-rr- 1 root root 1.2M 2011-02-21 10:47 table_task_meta-f-418-Data.db
rw-rr- 1 root root 18K 2011-02-21 10:47 table_task_meta-f-418-Filter.db
rw-rr- 1 root root 460K 2011-02-21 10:47 table_task_meta-f-418-Index.db
rw-rr- 1 root root 4.2K 2011-02-21 10:47 table_task_meta-f-418-Statistics.db
rw-rr- 1 root root 791K 2011-02-21 11:47 table_task_meta-f-419-Data.db
rw-rr- 1 root root 13K 2011-02-21 11:47 table_task_meta-f-419-Filter.db
rw-rr- 1 root root 311K 2011-02-21 11:47 table_task_meta-f-419-Index.db
rw-rr- 1 root root 4.2K 2011-02-21 11:47 table_task_meta-f-419-Statistics.db
rw-rr- 1 root root 57K 2011-02-21 12:11 table_task-tmp-f-1707-Filter.db
rw-rr- 1 root root 1.4M 2011-02-21 12:11 table_task-tmp-f-1707-Index.db
rw-rr- 1 root root 4.2K 2011-02-21 12:11 table_task-tmp-f-1707-Statistics.db

Cassandra log:

/software/cassandra/bin/cassandra
rm: cannot remove `/software/cassandra/lib/jna.jar': No such file or directory
root@intr1n3:/cassandra/data/table_task# INFO 12:47:29,020 Logging initialized
INFO 12:47:29,030 Heap size: 2614493184/2614493184
INFO 12:47:29,031 JNA not found. Native methods will be disabled.
INFO 12:47:29,038 Loading settings from file:/software/cassandra/conf/cassandra.yaml
INFO 12:47:29,320 DiskAccessMode is standard, indexAccessMode is mmap
INFO 12:47:29,332 Creating new commitlog segment /hd1/cassandra_md5/commitlog/CommitLog-1298288849332.log
INFO 12:47:29,422 Opening /cassandra/data/system/Schema-f-244
INFO 12:47:29,434 Opening /cassandra/data/system/Migrations-f-244
INFO 12:47:29,437 Opening /cassandra/data/system/LocationInfo-f-137
INFO 12:47:29,440 Opening /cassandra/data/system/HintsColumnFamily-f-352
INFO 12:47:29,441 Opening /cassandra/data/system/HintsColumnFamily-f-353
INFO 12:47:29,465 Loading schema version 54bc134e-2229-11e0-9159-fdf0b6b4b562
WARN 12:47:29,623 Schema definitions were defined both locally and in cassandra.yaml. Definitions in cassandra.yaml were ignored.
ERROR 12:47:29,638 Exception encountered during startup.
java.io.IOError: java.io.IOException: Failed to delete /cassandra/data/table_task/table_task-tmp-f-1707-Data.db
at org.apache.cassandra.io.sstable.SSTable.delete(SSTable.java:145)
at org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:468)
at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:153)
at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:316)
at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:79)
Caused by: java.io.IOException: Failed to delete /cassandra/data/table_task/table_task-tmp-f-1707-Data.db
at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:51)
at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:41)
at org.apache.cassandra.io.sstable.SSTable.delete(SSTable.java:133)
... 4 more
Exception encountered during startup.
java.io.IOError: java.io.IOException: Failed to delete /cassandra/data/table_task/table_task-tmp-f-1707-Data.db
at org.apache.cassandra.io.sstable.SSTable.delete(SSTable.java:145)
at org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:468)
at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:153)
at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:316)
at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:79)
Caused by: java.io.IOException: Failed to delete /cassandra/data/table_task/table_task-tmp-f-1707-Data.db
at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:51)
at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:41)
at org.apache.cassandra.io.sstable.SSTable.delete(SSTable.java:133)
... 4 more