Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-7007

Streaming Errors due to RejectedExecutionException

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Urgent
    • Resolution: Fixed
    • Fix Version/s: 2.0.7
    • Component/s: None
    • Labels:
      None
    • Severity:
      Critical

      Description

      With the same repair setup as CASSANDRA-6984, I'm now seeing repair failures due to some streaming tasks hitting RejectedExecutionException:

      ERROR [STREAM-OUT-/127.0.0.3] 2014-04-08 13:50:15,424 StreamSession.java (line 420) [Stream #9f64e230-bf4e-11e3-b994-43eb3a328df9] Streaming error occurred
      java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@1eb6b48a rejected from java.util.concurrent.ScheduledThreadPoolExecutor@6d06dfc6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
          at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
          at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
          at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:325)
          at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:530)
          at java.util.concurrent.Executors$DelegatedScheduledExecutorService.schedule(Executors.java:689)
          at org.apache.cassandra.streaming.StreamTransferTask.scheduleTimeout(StreamTransferTask.java:125)
          at org.apache.cassandra.streaming.StreamSession.fileSent(StreamSession.java:468)
          at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:60)
          at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:42)
          at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:45)
          at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:383)
          at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:355)
          at java.lang.Thread.run(Thread.java:724)
       INFO [STREAM-OUT-/127.0.0.3] 2014-04-08 13:50:15,431 StreamResultFuture.java (line 186) [Stream #9f64e230-bf4e-11e3-b994-43eb3a328df9] Session with /127.0.0.3 is complete
       WARN [STREAM-OUT-/127.0.0.3] 2014-04-08 13:50:15,431 StreamResultFuture.java (line 215) [Stream #9f64e230-bf4e-11e3-b994-43eb3a328df9] Stream failed
      ERROR [NonPeriodicTasks:1] 2014-04-08 13:50:15,449 CassandraDaemon.java (line 198) Exception in thread Thread[NonPeriodicTasks:1,5,main]
      java.lang.RuntimeException: Outgoing stream handler has been closed
          at org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:170)
          at org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:620)
          at org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:566)
          at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:120)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
          at java.util.concurrent.FutureTask.run(FutureTask.java:262)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:724)
      

      Since the ThreadPoolExecutor says its state is TERMINATED, I'm guessing this is due to a race condition where the timeout executor can be shutdown too early.

        Attachments

        1. 7007-2.0.txt
          1 kB
          Yuki Morishita

          Activity

            People

            • Assignee:
              yukim Yuki Morishita
              Reporter:
              thobbs Tom Hobbs
              Authors:
              Yuki Morishita
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: