[CASSANDRA-14826] cassandra spinning forever on 1 thread while initializing keyspace

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Duplicate
• Fix Version/s: 3.0.23, 3.11.9, 4.1-alpha1, 4.1
• Component/s: Local/Startup and Shutdown
• Severity: Normal
• Assignee: Marcus Eriksson
• Reporter: ingard mevåg

Authors

Marcus Eriksson

• Created: 16/Oct/18 13:01
• Updated: 16/Dec/22 21:56
• Resolved: 16/Sep/20 09:14

Description

When starting cassandra 3.0.17 it takes a long time to initialize a keyspace.

top shows 1 thread spinning at 100% cpu. Thread dump shows:

"main" - Thread t@1

    java.lang.Thread.State: RUNNABLE

         at java.util.TimSort.mergeHi(TimSort.java:850)

         at java.util.TimSort.mergeAt(TimSort.java:516)

         at java.util.TimSort.mergeCollapse(TimSort.java:441)

         at java.util.TimSort.sort(TimSort.java:245)

         at java.util.Arrays.sort(Arrays.java:1512)

         at java.util.ArrayList.sort(ArrayList.java:1454)

         at java.util.Collections.sort(Collections.java:175)

         at org.apache.cassandra.db.compaction.LeveledManifest.canAddSSTable(LeveledManifest.java:243)

         at org.apache.cassandra.db.compaction.LeveledManifest.add(LeveledManifest.java:146)

         - locked <c89b563> (a org.apache.cassandra.db.compaction.LeveledManifest)

         at org.apache.cassandra.db.compaction.LeveledCompactionStrategy.addSSTable(LeveledCompactionStrategy.java:298)

         at org.apache.cassandra.db.compaction.CompactionStrategyManager.startup(CompactionStrategyManager.java:135)

         at org.apache.cassandra.db.compaction.CompactionStrategyManager.reload(CompactionStrategyManager.java:187)

         - locked <742bfce7> (a org.apache.cassandra.db.compaction.CompactionStrategyManager)

         at org.apache.cassandra.db.compaction.CompactionStrategyManager.<init>(CompactionStrategyManager.java:75)

         at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:408)

         at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:363)

         at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:579)

         - locked <4e4f20d2> (a java.lang.Class)

         at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:556)

         at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:368)

         at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:305)

         at org.apache.cassandra.db.Keyspace.open(Keyspace.java:129)

         - locked <5318346c> (a java.lang.Class)

         at org.apache.cassandra.db.Keyspace.open(Keyspace.java:106)

         at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:262)

         at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:569)

         at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:697)

Issue Links

• duplicates: CASSANDRA-14103 Fix potential race during compaction strategy reload
• is duplicated by: CASSANDRA-14069 Node stopped serving write requests when a table has a lot of sstables