Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-3481

During repair, "incorrect data size" & "Connection reset" errors. Repair unable to complete.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Fix Version/s: 1.0.3
    • Component/s: None
    • Labels:
    • Severity:
      Normal

      Description

      This has been happening since 1.0.2. I wasn't on 1.0 for very long but I'm fairly certain repair was working ok. Repair worked decently for me in 0.8 (data bloat sucked). All my SSTables are version h.

      On one node:

      java.lang.AssertionError: incorrect row data size 596045 written to /mnt/cassandra/data/TRProd/Metrics1m-tmp-h-25036-Data.db; correct is 586675
      at org.apache.cassandra.io.sstable.SSTableWriter.appendFromStream(SSTableWriter.java:253)
      at org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:146)
      at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:87)
      at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:184)
      at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:81)

      On the other node:

      4999 - 0%, /mnt/cassandra/data/TRProd/Metrics1m-h-24953-Data.db sections=1707 progress=0/1513497639 - 0%, /mnt/cassandra/data/TRProd/Metrics1m-h-25000-Data.db sections=635 progress=0/53400713 - 0%, /mnt/cassandra/data/TRProd/Metrics1m-h-25002-Data.db sections=570 progress=0/709993 - 0%, /mnt/cassandra/data/TRProd/Metrics1m-h-25003-Data.db sections=550 progress=0/449498 - 0%, /mnt/cassandra/data/TRProd/Metrics1m-h-25005-Data.db sections=516 progress=0/316301 - 0%], 6 sstables.
      INFO [StreamStage:1] 2011-11-09 19:45:22,795 StreamOutSession.java (line 203) Streaming to /10.38.69.192
      ERROR [Streaming:1] 2011-11-09 19:47:47,964 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[Streaming:1,1,main]
      java.lang.RuntimeException: java.net.SocketException: Connection reset
      at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:619)
      Caused by: java.net.SocketException: Connection reset
      at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
      at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
      at com.ning.compress.lzf.ChunkEncoder.encodeAndWriteChunk(ChunkEncoder.java:133)
      at com.ning.compress.lzf.LZFOutputStream.writeCompressedBlock(LZFOutputStream.java:203)
      at com.ning.compress.lzf.LZFOutputStream.write(LZFOutputStream.java:97)
      at org.apache.cassandra.streaming.FileStreamTask.write(FileStreamTask.java:181)
      at org.apache.cassandra.streaming.FileStreamTask.stream(FileStreamTask.java:145)
      at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
      at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
      ... 3 more
      ERROR [Streaming:1] 2011-11-09 19:47:47,970 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[Streaming:1,1,main]
      java.lang.RuntimeException: java.net.SocketException: Connection reset
      at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:619)
      Caused by: java.net.SocketException: Connection reset
      at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
      at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
      at com.ning.compress.lzf.ChunkEncoder.encodeAndWriteChunk(ChunkEncoder.java:133)
      at com.ning.compress.lzf.LZFOutputStream.writeCompressedBlock(LZFOutputStream.java:203)
      at com.ning.compress.lzf.LZFOutputStream.write(LZFOutputStream.java:97)
      at org.apache.cassandra.streaming.FileStreamTask.write(FileStreamTask.java:181)
      at org.apache.cassandra.streaming.FileStreamTask.stream(FileStreamTask.java:145)
      at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
      at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
      ... 3 more

        Attachments

        1. 3481.patch
          36 kB
          Sylvain Lebresne
        2. 3481-v2.patch
          34 kB
          Sylvain Lebresne

          Activity

            People

            • Assignee:
              slebresne Sylvain Lebresne
              Reporter:
              efalcao Eric Falcao
              Authors:
              Sylvain Lebresne
              Reviewers:
              Jonathan Ellis
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: