Details
-
Bug
-
Status: Resolved
-
Normal
-
Resolution: Duplicate
-
None
-
None
-
None
-
Ubuntu
-
Normal
Description
A node is suffering from CASSANDRA-2793. Scrub is run to try and fix the problem. Although ulimit shows unlimited file handles allowed scrub eventually fails with java.io.FileNotFoundException / Too many open files:
INFO [CompactionExecutor:1] 2011-06-17 00:46:26,115 CompactionManager.java (line 511) Scrubbing SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7018-Data.db')
INFO [CompactionExecutor:1] 2011-06-17 00:46:26,225 CompactionManager.java (line 652) Scrub of SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7018-Data.db') complete: 275 rows in new sstable and 0 empty (tombstoned) rows dropped
INFO [CompactionExecutor:1] 2011-06-17 00:46:26,226 CompactionManager.java (line 511) Scrubbing SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7580-Data.db')
INFO [CompactionExecutor:1] 2011-06-17 00:46:28,383 CompactionManager.java (line 652) Scrub of SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7580-Data.db') complete: 297 rows in new sstable and 0 empty (tombstoned) rows dropped
INFO [CompactionExecutor:1] 2011-06-17 00:46:28,384 CompactionManager.java (line 511) Scrubbing SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7574-Data.db')
INFO [CompactionExecutor:1] 2011-06-17 00:46:29,300 CompactionManager.java (line 652) Scrub of SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7574-Data.db') complete: 347 rows in new sstable and 0 empty (tombstoned) rows dropped
INFO [CompactionExecutor:1] 2011-06-17 00:46:29,300 CompactionManager.java (line 511) Scrubbing SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7010-Data.db')
ERROR [CompactionExecutor:1] 2011-06-17 00:46:29,374 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.io.FileNotFoundException: /var/opt/cassandra/data/FightMyMonster/UserMonsters-tmp-f-7823-Data.db (Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:113)
at org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:78)
at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:2243)
at org.apache.cassandra.db.CompactionManager.maybeCreateWriter(CompactionManager.java:794)
at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:534)
at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
This error eventually "takes over":
ERROR [ReadStage:328] 2011-06-17 00:46:38,634 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[ReadStage:328,5,main]
java.io.IOError: java.io.FileNotFoundException: /var/opt/cassandra/data/FightMyMonster/MonsterMarket_3-f-3466-Data.db (Too many open files)
at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:78)
at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:553)
at org.apache.cassandra.db.RowIteratorFactory.getIterator(RowIteratorFactory.java:95)
at org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1442)
at org.apache.cassandra.service.RangeSliceVerbHandler.doVerb(RangeSliceVerbHandler.java:49)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.FileNotFoundException: /var/opt/cassandra/data/FightMyMonster/MonsterMarket_3-f-3466-Data.db (Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:113)
at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:108)
at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:93)
at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:74)
... 8 more
ERROR [ReadStage:343] 2011-06-17 00:46:48,624 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[ReadStage:343,5,main]
java.io.IOError: java.io.FileNotFoundException: /var/opt/cassandra/data/FightMyMonster/MonsterMarket_3-f-3468-Data.db (Too many open files)
at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:78)
at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:553)
at org.apache.cassandra.db.RowIteratorFactory.getIterator(RowIteratorFactory.java:95)
at org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1442)
at org.apache.cassandra.service.RangeSliceVerbHandler.doVerb(RangeSliceVerbHandler.java:49)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.FileNotFoundException: /var/opt/cassandra/data/FightMyMonster/MonsterMarket_3-f-3468-Data.db (Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:113)
at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:108)
at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:93)
at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:74)
... 8 more