[CASSANDRA-2134] Error in ThreadPoolExecutor

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 0.7.1

Environment

Linux 2.6.32-bpo.4edois1-openvz-amd64 #1 SMP x86_64 GNU/Linux

• Severity: Normal
• Assignee: Sylvain Lebresne Assign to me
• Reporter: Patrik Modesto

Authors

Sylvain Lebresne

Reviewers

Jonathan Ellis

• Created: 08/Feb/11 06:57
• Updated: 16/Apr/19 09:33
• Resolved: 09/Feb/11 14:23

Description

On my two-node test setup I get repeatedly following error:

The 10.0.18.129 server log:

 
 INFO 14:10:37,707 Node /10.0.18.99 has restarted, now UP again
 INFO 14:10:37,708 Checking remote schema before delivering hints
 INFO 14:10:37,708 Sleeping 45506ms to stagger hint delivery
 INFO 14:10:37,709 Node /10.0.18.99 state jump to normal
 INFO 14:11:23,215 Started hinted handoff for endpoint /10.0.18.99
ERROR 14:11:23,884 Error in ThreadPoolExecutor
java.lang.RuntimeException: java.lang.IllegalArgumentException
       at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
       at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.IllegalArgumentException
       at java.nio.Buffer.position(Buffer.java:218)
       at org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:117)
       at org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:111)
       at org.apache.cassandra.db.HintedHandOffManager.getTableAndCFNames(HintedHandOffManager.java:237)
       at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:306)
       at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:88)
       at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:385)
       at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
       ... 3 more
ERROR 14:11:23,885 Fatal exception in thread Thread[HintedHandoff:1,1,main]
java.lang.RuntimeException: java.lang.IllegalArgumentException
       at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
       at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.IllegalArgumentException
       at java.nio.Buffer.position(Buffer.java:218)
       at org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:117)
       at org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:111)
       at org.apache.cassandra.db.HintedHandOffManager.getTableAndCFNames(HintedHandOffManager.java:237)
       at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:306)
       at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:88)
       at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:385)
       at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
       ... 3 more

The 10.0.18.99 server log:

 
 INFO 14:10:37,691 Binding thrift service to /0.0.0.0:9160
 INFO 14:10:37,693 Using TFastFramedTransport with a max frame size of
15728640 bytes.
 INFO 14:10:37,695 Listening for thrift clients...
 INFO 14:10:38,337 GC for ParNew: 954 ms, 658827608 reclaimed leaving
966732432 used; max is 4265607168
 INFO 14:11:27,142 Started hinted handoff for endpoint /10.0.18.129
ERROR 14:11:27,370 Error in ThreadPoolExecutor
java.lang.RuntimeException: java.lang.IllegalArgumentException
       at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
       at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.IllegalArgumentException
       at java.nio.Buffer.position(Buffer.java:218)
       at org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:117)
       at org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:111)
       at org.apache.cassandra.db.HintedHandOffManager.getTableAndCFNames(HintedHandOffManager.java:237)
       at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:306)
       at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:88)
       at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:385)
       at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
       ... 3 more
ERROR 14:11:27,371 Fatal exception in thread Thread[HintedHandoff:1,1,main]
java.lang.RuntimeException: java.lang.IllegalArgumentException
       at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
       at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.IllegalArgumentException
       at java.nio.Buffer.position(Buffer.java:218)
       at org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:117)
       at org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:111)
       at org.apache.cassandra.db.HintedHandOffManager.getTableAndCFNames(HintedHandOffManager.java:237)
       at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:306)
       at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:88)
       at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:385)
       at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
       ... 3 more

It happen durring batch_mutate test or after restart, when there are commitlogs to replay. Using current 0.7.1 from cassandra-0.7 branch.