Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-3481

During repair, "incorrect data size" & "Connection reset" errors. Repair unable to complete.

Agile BoardAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Normal
    • Resolution: Fixed
    • 1.0.3
    • None
    • Normal

    Description

      This has been happening since 1.0.2. I wasn't on 1.0 for very long but I'm fairly certain repair was working ok. Repair worked decently for me in 0.8 (data bloat sucked). All my SSTables are version h.

      On one node:

      java.lang.AssertionError: incorrect row data size 596045 written to /mnt/cassandra/data/TRProd/Metrics1m-tmp-h-25036-Data.db; correct is 586675
      at org.apache.cassandra.io.sstable.SSTableWriter.appendFromStream(SSTableWriter.java:253)
      at org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:146)
      at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:87)
      at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:184)
      at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:81)

      On the other node:

      4999 - 0%, /mnt/cassandra/data/TRProd/Metrics1m-h-24953-Data.db sections=1707 progress=0/1513497639 - 0%, /mnt/cassandra/data/TRProd/Metrics1m-h-25000-Data.db sections=635 progress=0/53400713 - 0%, /mnt/cassandra/data/TRProd/Metrics1m-h-25002-Data.db sections=570 progress=0/709993 - 0%, /mnt/cassandra/data/TRProd/Metrics1m-h-25003-Data.db sections=550 progress=0/449498 - 0%, /mnt/cassandra/data/TRProd/Metrics1m-h-25005-Data.db sections=516 progress=0/316301 - 0%], 6 sstables.
      INFO [StreamStage:1] 2011-11-09 19:45:22,795 StreamOutSession.java (line 203) Streaming to /10.38.69.192
      ERROR [Streaming:1] 2011-11-09 19:47:47,964 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[Streaming:1,1,main]
      java.lang.RuntimeException: java.net.SocketException: Connection reset
      at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:619)
      Caused by: java.net.SocketException: Connection reset
      at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
      at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
      at com.ning.compress.lzf.ChunkEncoder.encodeAndWriteChunk(ChunkEncoder.java:133)
      at com.ning.compress.lzf.LZFOutputStream.writeCompressedBlock(LZFOutputStream.java:203)
      at com.ning.compress.lzf.LZFOutputStream.write(LZFOutputStream.java:97)
      at org.apache.cassandra.streaming.FileStreamTask.write(FileStreamTask.java:181)
      at org.apache.cassandra.streaming.FileStreamTask.stream(FileStreamTask.java:145)
      at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
      at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
      ... 3 more
      ERROR [Streaming:1] 2011-11-09 19:47:47,970 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[Streaming:1,1,main]
      java.lang.RuntimeException: java.net.SocketException: Connection reset
      at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:619)
      Caused by: java.net.SocketException: Connection reset
      at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
      at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
      at com.ning.compress.lzf.ChunkEncoder.encodeAndWriteChunk(ChunkEncoder.java:133)
      at com.ning.compress.lzf.LZFOutputStream.writeCompressedBlock(LZFOutputStream.java:203)
      at com.ning.compress.lzf.LZFOutputStream.write(LZFOutputStream.java:97)
      at org.apache.cassandra.streaming.FileStreamTask.write(FileStreamTask.java:181)
      at org.apache.cassandra.streaming.FileStreamTask.stream(FileStreamTask.java:145)
      at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
      at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
      ... 3 more

      Attachments

        1. 3481.patch
          36 kB
          Sylvain Lebresne
        2. 3481-v2.patch
          34 kB
          Sylvain Lebresne

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            slebresne Sylvain Lebresne Assign to me
            efalcao Eric Falcao
            Sylvain Lebresne
            Jonathan Ellis
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment