[CASSANDRA-1732] nodetool move throws Assertion Error

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 0.7.0 rc 1

Environment

CentOS 5.4
Cassandra Trunk SVN Revision #1034158

• Severity: Normal
• Assignee: T Jake Luciani
• Reporter: Arya Goudarzi

Authors

T Jake Luciani

Reviewers

Jonathan Ellis

• Created: 12/Nov/10 00:34
• Updated: 16/Apr/19 09:33
• Resolved: 19/Nov/10 14:34

Description

Started from a clean slate 3 node cluster. I first started 1 node and bootstrapped the second and third node into the cluster. I created some Keyspaces and inserted some test data, I ended up with this ring:

[agoudarzi@cas-test1 ~]$ nodetool --host localhost ring
Address Status State Load Token
142685436305748685139980028665762955655
10.50.26.133 Up Normal 160.51 KB 57614844575514069274136376807820902791
10.50.26.134 Up Normal 160.51 KB 100150140440631377207058202736791929223
10.50.26.132 Up Normal 165.48 KB 142685436305748685139980028665762955655

Now I wanted to test manual moving nodes to balanced tokens:
stage1:agoudarzi:~:$ python test.py 3
56713727820156410577229101238628035242
113427455640312821154458202477256070484
170141183460469231731687303715884105727

So I did nodetool move on 10.50.26.132:
[agoudarzi@cas-test1 ~]$ nodetool --host localhost move 56713727820156410577229101238628035242

All went fine.
[agoudarzi@cas-test1 ~]$ nodetool --host localhost ring
Address Status State Load Token
100150140440631377207058202736791929223
10.50.26.132 Up Normal 603.03 KB 56713727820156410577229101238628035242
10.50.26.133 Up Normal 15.18 MB 57614844575514069274136376807820902791
10.50.26.134 Up Normal 15.19 MB 100150140440631377207058202736791929223

Now I wanted to move the second node 10.50.26.133:

[agoudarzi@cas-test2 ~]$ nodetool --host localhost move 113427455640312821154458202477256070484
Exception in thread "main" java.lang.AssertionError
at org.apache.cassandra.service.StorageService.getLocalToken(StorageService.java:1128)
at org.apache.cassandra.service.StorageService.startLeaving(StorageService.java:1527)
at org.apache.cassandra.service.StorageService.move(StorageService.java:1666)
at org.apache.cassandra.service.StorageService.move(StorageService.java:1641)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1449)
at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1284)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1382)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
at sun.rmi.transport.Transport$1.run(Transport.java:177)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)

I am attacking the logs for my 3 nodes. For your reference I refer to these IPs as these nodes:

node1: 10.50.26.132
node2: 10.50.26.133
node3: 10.50.26.134

I have seen similar exception being thrown in CASSANDRA-1670.

Please investigate.

-Arya