Details
-
Bug
-
Status: Resolved
-
Low
-
Resolution: Cannot Reproduce
-
None
-
None
-
Low
Description
I was unable to trigger compaction on one of the column families so I ran scrub on that CF.
After scrub when I tried to run compaction I got following error.
Error occured during compaction java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /mnt2/var/lib/cassandra/data/AudienceNetwork/Audience-g-9898-Data.db (Too many open files) at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.compaction.CompactionManager.performMajor(CompactionManager.java:277) at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1762) at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1358) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427) at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360) at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) at sun.rmi.transport.Transport$1.run(Transport.java:159) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:155) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOError: java.io.FileNotFoundException: /mnt2/var/lib/cassandra/data/AudienceNetwork/Audience-g-9898-Data.db (Too many open files) at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61) at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:660) at org.apache.cassandra.db.compaction.CompactionIterator.getCollatingIterator(CompactionIterator.java:92) at org.apache.cassandra.db.compaction.CompactionIterator.<init>(CompactionIterator.java:68) at org.apache.cassandra.db.compaction.CompactionManager.doCompactionWithoutSizeEstimation(CompactionManager.java:552) at org.apache.cassandra.db.compaction.CompactionManager.doCompaction(CompactionManager.java:506) at org.apache.cassandra.db.compaction.CompactionManager$4.call(CompactionManager.java:319) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more Caused by: java.io.FileNotFoundException: /mnt2/var/lib/cassandra/data/AudienceNetwork/Audience-g-9898-Data.db (Too many open files) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212) at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:113) at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:57) ... 11 more
We are using cassndra-0.8.2 and I have set the ulimit to 1048576 on Ubuntu Lucid system.
Also getting the same error while scrub.