Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-1732

nodetool move throws Assertion Error

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Fix Version/s: 0.7.0 rc 1
    • Component/s: None
    • Labels:
      None
    • Environment:

      CentOS 5.4
      Cassandra Trunk SVN Revision #1034158

    • Severity:
      Normal

      Description

      Started from a clean slate 3 node cluster. I first started 1 node and bootstrapped the second and third node into the cluster. I created some Keyspaces and inserted some test data, I ended up with this ring:

      [agoudarzi@cas-test1 ~]$ nodetool --host localhost ring
      Address Status State Load Token
      142685436305748685139980028665762955655
      10.50.26.133 Up Normal 160.51 KB 57614844575514069274136376807820902791
      10.50.26.134 Up Normal 160.51 KB 100150140440631377207058202736791929223
      10.50.26.132 Up Normal 165.48 KB 142685436305748685139980028665762955655

      Now I wanted to test manual moving nodes to balanced tokens:
      stage1:agoudarzi:~:$ python test.py 3
      56713727820156410577229101238628035242
      113427455640312821154458202477256070484
      170141183460469231731687303715884105727

      So I did nodetool move on 10.50.26.132:
      [agoudarzi@cas-test1 ~]$ nodetool --host localhost move 56713727820156410577229101238628035242

      All went fine.
      [agoudarzi@cas-test1 ~]$ nodetool --host localhost ring
      Address Status State Load Token
      100150140440631377207058202736791929223
      10.50.26.132 Up Normal 603.03 KB 56713727820156410577229101238628035242
      10.50.26.133 Up Normal 15.18 MB 57614844575514069274136376807820902791
      10.50.26.134 Up Normal 15.19 MB 100150140440631377207058202736791929223

      Now I wanted to move the second node 10.50.26.133:

      [agoudarzi@cas-test2 ~]$ nodetool --host localhost move 113427455640312821154458202477256070484
      Exception in thread "main" java.lang.AssertionError
      at org.apache.cassandra.service.StorageService.getLocalToken(StorageService.java:1128)
      at org.apache.cassandra.service.StorageService.startLeaving(StorageService.java:1527)
      at org.apache.cassandra.service.StorageService.move(StorageService.java:1666)
      at org.apache.cassandra.service.StorageService.move(StorageService.java:1641)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:616)
      at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
      at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
      at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
      at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
      at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
      at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
      at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1449)
      at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
      at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1284)
      at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1382)
      at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:616)
      at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
      at sun.rmi.transport.Transport$1.run(Transport.java:177)
      at java.security.AccessController.doPrivileged(Native Method)
      at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
      at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
      at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
      at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
      at java.lang.Thread.run(Thread.java:636)

      I am attacking the logs for my 3 nodes. For your reference I refer to these IPs as these nodes:

      node1: 10.50.26.132
      node2: 10.50.26.133
      node3: 10.50.26.134

      I have seen similar exception being thrown in CASSANDRA-1670.

      Please investigate.

      -Arya

        Attachments

        1. 1732_v1.txt
          0.7 kB
          T Jake Luciani
        2. node1.log
          52 kB
          Arya Goudarzi
        3. node2.log
          45 kB
          Arya Goudarzi
        4. node3.log
          43 kB
          Arya Goudarzi

          Activity

            People

            • Assignee:
              tjake T Jake Luciani
              Reporter:
              arya Arya Goudarzi
              Authors:
              T Jake Luciani
              Reviewers:
              Jonathan Ellis
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: