[CASSANDRA-4847] Bad disk causes death of node despite disk_failure_policy

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 1.2.1
• Severity: Normal
• Assignee: Kirk True
• Reporter: Kirk True

Authors

Kirk True

Reviewers

Jonathan Ellis

• Created: 22/Oct/12 18:11
• Updated: 16/Apr/19 09:32
• Resolved: 13/Dec/12 16:20

Description

Steps:

1. Create a bad disk via device mapper
2. Specify good disk and bad disk is data directory
3. Set disk_failure_policy to best_effort in cassandra.yaml
4. Start node

Expected:

Attempts to create system directories to fail (as expected) on bad disk, and have it added to blacklisted directories.

Actual:

Node start up aborts due to uncaught error:

FSWriteError in /mnt/bad_disk/system_traces/sessions
        at org.apache.cassandra.io.util.FileUtils.createDirectory(FileUtils.java:258)
        at org.apache.cassandra.db.Directories.<init>(Directories.java:104)
        at org.apache.cassandra.db.Directories.create(Directories.java:90)
        at org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:404)
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:227)
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:393)
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:436)
Caused by: java.io.IOException: Failed to mkdirs /mnt/bad_disk/system_traces/sessions
        ... 7 more