Details
-
Bug
-
Status: Resolved
-
Normal
-
Resolution: Cannot Reproduce
-
None
-
None
-
None
-
ubuntu
sun-java6 6.24-1build0.10.10.1
-
Normal
Description
Upgraded a single node in my ring to 1.1.7. Upgrade process went normally with no errors. However, as soon as the node joined the ring, it started spewing this exception hundreds of times a second:
WARN [ReadStage:22] 2012-12-12 02:00:56,181 FileUtils.java (line 116) Failed closing org.apache.cassandra.db.columniterator.SSTableSliceIterator@5959baa2 java.io.IOException: Bad file descriptor at sun.nio.ch.FileDispatcher.preClose0(Native Method) at sun.nio.ch.FileDispatcher.preClose(FileDispatcher.java:59) at sun.nio.ch.FileChannelImpl.implCloseChannel(FileChannelImpl.java:96) at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:97) at java.io.FileInputStream.close(FileInputStream.java:258) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.close(CompressedRandomAccessReader.java:131) at sun.nio.ch.FileChannelImpl.implCloseChannel(FileChannelImpl.java:121) at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:97) at java.io.RandomAccessFile.close(RandomAccessFile.java:541) at org.apache.cassandra.io.util.RandomAccessReader.close(RandomAccessReader.java:224) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.close(CompressedRandomAccessReader.java:130) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.close(SSTableSliceIterator.java:132) at org.apache.cassandra.io.util.FileUtils.closeQuietly(FileUtils.java:112) at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:300) at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1347) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1209) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1144) at org.apache.cassandra.db.Table.getRow(Table.java:378) at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69) at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:51) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662)
The node was not responding to reads on any CFs, so I was forced to do an emergency roll-back and abandon the upgrade.
Node has roughly 3800 sstables. Both LCS and SizeTiered, as well as compressed and uncompressed CFs.
After some digging on a test node, I've determined that the issue occurs when attempting to read/upgrade/scrub a compressed 1.0.11-generated sstable on 1.1.7.