[CASSANDRA-11981] Cassandra 2.2 -> 3.5 upgradesstables results in error: null

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Information Provided

Environment

# java -version
openjdk version "1.8.0_91"
OpenJDK Runtime Environment (build 1.8.0_91-b14)
OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)

# uname -a
Linux cassandra-dfs-10-10-160-19 4.4.11-23.53.amzn1.x86_64 #1 SMP Wed Jun 1 22:22:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

# rpm -qa datastax-ddc
datastax-ddc-3.5.0-1.noarch

• Severity: Normal
• Assignee: Unassigned
• Reporter: Victor Trac
• Created: 08/Jun/16 21:18
• Updated: 16/Apr/19 09:30
• Resolved: 19/Nov/18 02:25

Description

We upgraded from Cassandra 2.2.1 -> 3.5 on a 6-node cluster. I ran 'nodetool upgradesstables' on all nodes successfully except for one. It would take a 2-3 days ('nodetool netstats` showed continual progress), but then the server would error out with a java.io.EOFException. After trying upgradesstables several times, I tried a scrub. Again, it ran for a couple of days and then errored out with a very similar error:

# nodetool scrub
error: null
-- StackTrace --
java.io.EOFException
        at java.io.DataInputStream.readByte(DataInputStream.java:267)
        at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:215)
        at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
        at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source)
        at javax.management.remote.rmi.RMIConnectionImpl_Stub.invoke(Unknown Source)
        at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:1020)
        at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:298)
        at com.sun.proxy.$Proxy7.scrub(Unknown Source)
        at org.apache.cassandra.tools.NodeProbe.scrub(NodeProbe.java:248)
        at org.apache.cassandra.tools.NodeProbe.scrub(NodeProbe.java:280)
        at org.apache.cassandra.tools.nodetool.Scrub.execute(Scrub.java:66)
        at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:250)
        at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:164)