[CASSANDRA-6398] nodetool removenode error 'Endpoint /x.x.x.x generation changed while trying to remove it' occurs regularly.

• Type: Bug
• Status: Resolved
• Priority: Low
• Resolution: Cannot Reproduce
• Fix Version/s: 2.0.5
• Component/s: Tool/nodetool
• Severity: Low
• Assignee: Tom Hobbs
• Reporter: Ryan McGuire

Authors

Tom Hobbs

Reviewers

Brandon Williams

• Created: 22/Nov/13 18:06
• Updated: 16/Apr/19 09:31
• Resolved: 02/Jan/14 22:17

Description

I see this error somewhat regularly when running a nodetool removenode command. The command completes successfully, ie, the node is removed, so I'm not sure what this message is telling me.

$ nodetool -p 7100 removenode bff9072e-4bb6-42fa-937c-bb73bcc094bc

Exception in thread "main" java.lang.RuntimeException: Endpoint /127.0.0.2 generation changed while trying to remove it
	at org.apache.cassandra.gms.Gossiper.advertiseRemoving(Gossiper.java:421)
	at org.apache.cassandra.service.StorageService.removeNode(StorageService.java:3080)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
	at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:235)
	at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
	at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:250)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:791)
	at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1486)
	at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:96)
	at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1327)
	at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1419)
	at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:847)
	at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
	at sun.rmi.transport.Transport$1.run(Transport.java:177)
	at sun.rmi.transport.Transport$1.run(Transport.java:174)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:722)