Details
-
Bug
-
Status: Resolved
-
Low
-
Resolution: Cannot Reproduce
-
None
-
None
-
Low
Description
This happened using 0.6-beta3 on a test two nodes cluster.
Steps that lead to problem:
- launch node A
- load data in A
- launch node B connected to A
- load data in both (replicationfactor is 1)
- use nodetool to decommission A
At this point something went wrong inside A, and the command seemingly failed:
rff@A$ ./bin/nodetool --host localhost decommission
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
at $Proxy0.decommission(Unknown Source)
at org.apache.cassandra.tools.NodeProbe.decommission(NodeProbe.java:324)
at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:447)
Caused by: java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is:
java.io.EOFException
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:209)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:142)
at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl_Stub.invoke(Unknown Source)
at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:993)
at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:288)
... 3 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readByte(DataInputStream.java:250)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:195)
... 8 more
At this point, `nodetool streams` on A reported Mode:decommissioned but still sending streams.
The same way, node B still reported Mode: normal and still receiving streams .
In both case the streaming values where reported as 0/size-of-data for all the files.
Having turned off node A, after ~24 hours, node B still reports the same thing.
The decommissioning seems to have actually worked, but the status never got updated.