Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-5717

Repair causes streaming errors

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Duplicate
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      CentOS release 6.3 (Final)

    • Severity:
      Normal

      Description

      I've changed the replication factor for one of keyspaces and now I'm running repairs on column families (manually). I have 3 nodes cluster. Here is the error in the error log for one of the nodes.

      Error Log
      ERROR [Streaming to /NODE_IP:1] 2013-07-01 09:31:29,819 CassandraDaemon.java (line 192) Exception in thread Thread[Streaming to /NODE_IP:1,5,main]
      java.lang.RuntimeException: java.io.IOException: Broken pipe
              at com.google.common.base.Throwables.propagate(Throwables.java:160)
              at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
              at java.lang.Thread.run(Unknown Source)
      Caused by: java.io.IOException: Broken pipe
              at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
              at sun.nio.ch.FileChannelImpl.transferToDirectly(Unknown Source)
              at sun.nio.ch.FileChannelImpl.transferTo(Unknown Source)
              at org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:93)
              at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
              at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
              ... 3 more
      ERROR [Streaming to /NODE_IP:2] 2013-07-01 09:44:18,372 CassandraDaemon.java (line 192) Exception in thread Thread[Streaming to /NODE_IP:2,5,main]
      java.lang.RuntimeException: java.io.IOException: Broken pipe
              at com.google.common.base.Throwables.propagate(Throwables.java:160)
              at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
              at java.lang.Thread.run(Unknown Source)
      Caused by: java.io.IOException: Broken pipe
              at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
              at sun.nio.ch.FileChannelImpl.transferToDirectly(Unknown Source)
              at sun.nio.ch.FileChannelImpl.transferTo(Unknown Source)
              at org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:93)
              at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
              at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
              ... 3 more
      
      netstats for one of the nodes
      Mode: NORMAL
      Not sending any streams.
      Streaming from: /IP
         KEYSPACE: /var/lib/cassandra/data/KEYSPACE/ns_history_org/KEYSPACE-ns_history_org-hf-282-Data.db sections=1 progress=0/1393915753 - 0%
         KEYSPACE: /var/lib/cassandra/data/KEYSPACE/ns_history_org/KEYSPACE-ns_history_org-ic-455-Data.db sections=1 progress=0/792707 - 0%
      Streaming from: /IP
         KEYSPACE: /var/lib/cassandra/data/KEYSPACE/ns_history_org/KEYSPACE-ns_history_org-hf-255-Data.db sections=1 progress=0/1398197628 - 0%
         KEYSPACE: /var/lib/cassandra/data/KEYSPACE/ns_history_biz/KEYSPACE-ns_history_biz-ic-341-Data.db sections=1 progress=0/6153542 - 0%
         KEYSPACE: /var/lib/cassandra/data/KEYSPACE/listing/KEYSPACE-listing-hf-539-Data.db sections=1 progress=0/86968194 - 0%
         KEYSPACE: /var/lib/cassandra/data/KEYSPACE/ns_history_biz/KEYSPACE-ns_history_biz-hf-244-Data.db sections=1 progress=0/322197762 - 0%
         KEYSPACE: /var/lib/cassandra/data/KEYSPACE/ns_history_biz/KEYSPACE-ns_history_biz-ic-346-Data.db sections=1 progress=0/6219503 - 0%
         KEYSPACE: /var/lib/cassandra/data/KEYSPACE/listing/KEYSPACE-listing-hf-487-Data.db sections=1 progress=0/2689291466 - 0%
         KEYSPACE: /var/lib/cassandra/data/KEYSPACE/listing/KEYSPACE-listing-ic-684-Data.db sections=1 progress=0/3717513 - 0%
         KEYSPACE: /var/lib/cassandra/data/KEYSPACE/ns_history_org/KEYSPACE-ns_history_org-ic-413-Data.db sections=1 progress=0/22256993 - 0%
         KEYSPACE: /var/lib/cassandra/data/KEYSPACE/listing/KEYSPACE-listing-hf-529-Data.db sections=1 progress=0/345419053 - 0%
         KEYSPACE: /var/lib/cassandra/data/KEYSPACE/listing/KEYSPACE-listing-hf-534-Data.db sections=1 progress=0/88759930 - 0%
         KEYSPACE: /var/lib/cassandra/data/KEYSPACE/listing/KEYSPACE-listing-hf-509-Data.db sections=1 progress=0/365451892 - 0%
      Read Repair Statistics:
      Attempted: 10696592
      Mismatch (Blocking): 48873
      Mismatch (Background): 47308
      Pool Name                    Active   Pending      Completed
      Commands                        n/a         0      339083095
      Responses                       n/a         0      270274923
      

      Netstats showing zero progress and it's not progressing.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                nacholibre Yoan Arnaudov
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: