[CASSANDRA-4733] Last written key >= current key exception when streaming

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Cannot Reproduce
• Fix Version/s: 1.2.0 beta 2
• Severity: Normal
• Assignee: Yuki Morishita
• Reporter: Brandon Williams

Authors

Yuki Morishita

• Created: 28/Sep/12 17:28
• Updated: 16/Apr/19 09:32
• Resolved: 13/Nov/12 18:08

Description

ERROR 16:52:56,260 Exception in thread Thread[Streaming to /10.179.111.137:1,5,main]
java.lang.RuntimeException: java.io.IOException: Connection reset by peer
        at com.google.common.base.Throwables.propagate(Throwables.java:160)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcher.write0(Native Method)
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:72)
        at sun.nio.ch.IOUtil.write(IOUtil.java:43)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
        at java.nio.channels.Channels.writeFullyImpl(Channels.java:59)
        at java.nio.channels.Channels.writeFully(Channels.java:81)
        at java.nio.channels.Channels.access$000(Channels.java:47)
        at java.nio.channels.Channels$1.write(Channels.java:155)
        at com.ning.compress.lzf.ChunkEncoder.encodeAndWriteChunk(ChunkEncoder.java:133)
        at com.ning.compress.lzf.LZFOutputStream.writeCompressedBlock(LZFOutputStream.java:203)
        at com.ning.compress.lzf.LZFOutputStream.write(LZFOutputStream.java:97)
        at org.apache.cassandra.streaming.FileStreamTask.write(FileStreamTask.java:218)
        at org.apache.cassandra.streaming.FileStreamTask.stream(FileStreamTask.java:164)
        at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        ... 3 more
ERROR 16:53:03,951 Exception in thread Thread[Thread-11,5,main]
java.lang.RuntimeException: Last written key DecoratedKey(113424593524874987650593774422007331058, 3036303936343535) >= current key DecoratedKey(59229538317742990547810678738983628664, 3036313133373139) writing into /var/lib/cassandra/data/Keyspace1-Standard1-tmp-ia-95-Data.db
        at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:132)
        at org.apache.cassandra.io.sstable.SSTableWriter.appendFromStream(SSTableWriter.java:208)
        at org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:164)
        at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:107)
        at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:220)
        at org.apache.cassandra.net.IncomingTcpConnection.handleStream(IncomingTcpConnection.java:165)
        at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:65)

I didn't do anything fancy here, just inserted about 6M keys at rf=2, then ran repair and got this.