[CASSANDRA-14158] [DTEST] [TRUNK] repair_test.py::test_dead_coordinator is flaky due to JMX connection error from nodetool

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Component/s: Test/dtest/python
• Severity: Normal
• Assignee: Marcus Eriksson
• Reporter: Michael Kjellman

Authors

Marcus Eriksson

Reviewers

Sam Tunnicliffe

• Created: 10/Jan/18 07:16
• Updated: 16/Apr/19 09:29
• Resolved: 30/Jan/18 08:02

Description

repair_test.py::test_dead_coordinator is flaky due to occasionally failing when a JMX connection error is propagated by nodetool.

the test has failed 4+ times for the same reason.

latest failure can be found in the artifacts for the following circleci run:
https://circleci.com/gh/mkjellman/cassandra/538

I think that this might be expected behavior for this test and we just need to catch any ToolError exceptions thrown and only fail if included stack is for any error other than "JMX connection closed."

stderr: error: [2018-01-10 07:07:55,178] JMX connection closed. You should check server log for repair status of keyspace system_traces(Subsequent keyspaces are not going to be repaired).
-- StackTrace --
java.io.IOException: [2018-01-10 07:07:55,178] JMX connection closed. You should check server log for repair status of keyspace system_traces(Subsequent keyspaces are not going to be repaired).
	at org.apache.cassandra.tools.RepairRunner.handleConnectionFailed(RepairRunner.java:104)
	at org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:86)
	at javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:275)
	at javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:352)
	at javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:337)
	at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:248)
	at javax.management.remote.rmi.RMIConnector.sendNotification(RMIConnector.java:441)
	at javax.management.remote.rmi.RMIConnector.access$1200(RMIConnector.java:121)
	at javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1531)
	at javax.management.remote.rmi.RMIConnector$RMINotifClient.fetchNotifs(RMIConnector.java:1352)
	at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchOneNotif(ClientNotifForwarder.java:655)
	at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchNotifs(ClientNotifForwarder.java:607)
	at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:471)
	at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)
	at com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)

Issue Links

• links to: GitHub Pull Request #16