[CASSANDRA-8515] Commit log stop policy not enforced correctly during startup - ASF JIRA

Log work

Agile Board

Rank to Top

Rank to Bottom

Attach files

Attach Screenshot

Bulk Copy Attachments

Bulk Move Attachments

Voters

Watch issue

Watchers

Create sub-task

Convert to sub-task

Move

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Low
Resolution: Fixed
Fix Version/s: 2.1.9, 2.2.1, 3.0 beta 1
Component/s: Legacy/Coordination, Local/Startup and Shutdown
Labels:
None

Severity:
Low

Description

If the commit log directory has no free space, Cassandra hangs on startup.

The main thread is waiting:

"main" prio=9 tid=0x00007fefe400f800 nid=0x1303 waiting on condition [0x000000010b9c1000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000007dc8c5fc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.cassandra.db.commitlog.CommitLogAllocator.fetchSegment(CommitLogAllocator.java:137)
	at org.apache.cassandra.db.commitlog.CommitLog.activateNextSegment(CommitLog.java:299)
	at org.apache.cassandra.db.commitlog.CommitLog.<init>(CommitLog.java:73)
	at org.apache.cassandra.db.commitlog.CommitLog.<clinit>(CommitLog.java:53)
	at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:360)
	at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:339)
	at org.apache.cassandra.db.RowMutation.apply(RowMutation.java:211)
	at org.apache.cassandra.cql3.statements.ModificationStatement.executeInternal(ModificationStatement.java:699)
	at org.apache.cassandra.cql3.QueryProcessor.processInternal(QueryProcessor.java:208)
	at org.apache.cassandra.db.SystemKeyspace.updateSchemaVersion(SystemKeyspace.java:390)
	- locked <0x00000007de2f2ce0> (a java.lang.Class for org.apache.cassandra.db.SystemKeyspace)
	at org.apache.cassandra.config.Schema.updateVersion(Schema.java:384)
	at org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:532)
	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:270)
	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585)

but COMMIT-LOG-ALLOCATOR is RUNNABLE:

"COMMIT-LOG-ALLOCATOR" prio=9 tid=0x00007fefe5402800 nid=0x7513 in Object.wait() [0x0000000118252000]
   java.lang.Thread.State: RUNNABLE
	at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:116)
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
	at java.lang.Thread.run(Thread.java:745)

but making no progress.

This behaviour has changed since 1.2 (see ~~CASSANDRA-5737~~).