[CASSANDRA-11496] dtest failure in consistency_test.TestAvailability.test_network_topology_strategy

• Type: Improvement
• Status: Resolved
• Priority: Normal
• Resolution: Cannot Reproduce
• Labels: dtest
• Assignee: Russ Hatch
• Reporter: Michael Shuler

Authors

Russ Hatch

• Created: 04/Apr/16 22:35
• Updated: 16/Apr/19 09:30
• Resolved: 21/Apr/16 22:18

Description

example failure:

http://cassci.datastax.com/job/cassandra-2.1_large_dtest/2/testReport/consistency_test/TestAvailability/test_network_topology_strategy

Failed on CassCI build cassandra-2.1_large_dtest #2

Configured for m3.2xlarge instance - OOM starting node9:

Error Message

Error starting node9.
-------------------- >> begin captured logging << --------------------
dtest: DEBUG: cluster ccm directory: /mnt/tmp/dtest-SEXEwJ
dtest: DEBUG: Custom init_config not found. Setting defaults.
dtest: DEBUG: Done setting configuration options:
{   'initial_token': None,
    'num_tokens': '32',
    'phi_convict_threshold': 5,
    'range_request_timeout_in_ms': 10000,
    'read_request_timeout_in_ms': 10000,
    'request_timeout_in_ms': 10000,
    'truncate_request_timeout_in_ms': 10000,
    'write_request_timeout_in_ms': 10000}
--------------------- >> end captured logging << ---------------------
Stacktrace

  File "/usr/lib/python2.7/unittest/case.py", line 329, in run
    testMethod()
  File "/home/automaton/cassandra-dtest/consistency_test.py", line 316, in test_network_topology_strategy
    self._start_cluster()
  File "/home/automaton/cassandra-dtest/consistency_test.py", line 95, in _start_cluster
    cluster.populate(nodes).start(wait_for_binary_proto=True, wait_other_notice=True)
  File "/home/automaton/ccm/ccmlib/cluster.py", line 360, in start
    raise NodeError("Error starting {0}.".format(node.name), p)
"Error starting node9.\n-------------------- >> begin captured logging << --------------------\ndtest: DEBUG: cluster ccm directory: /mnt/tmp/dtest-SEXEwJ\ndtest: DEBUG: Custom init_config not found. Setting defaults.\ndtest: DEBUG: Done setting configuration options:\n{   'initial_token': None,\n    'num_tokens': '32',\n    'phi_convict_threshold': 5,\n    'range_request_timeout_in_ms': 10000,\n    'read_request_timeout_in_ms': 10000,\n    'request_timeout_in_ms': 10000,\n    'truncate_request_timeout_in_ms': 10000,\n    'write_request_timeout_in_ms': 10000}\n--------------------- >> end captured logging << ---------------------"
Standard Output

[node9 ERROR] java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:714)
	at org.apache.cassandra.net.OutboundTcpConnectionPool.start(OutboundTcpConnectionPool.java:174)
	at org.apache.cassandra.net.MessagingService.getConnectionPool(MessagingService.java:548)
	at org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:557)
	at org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:706)
	at org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:675)
	at org.apache.cassandra.gms.Gossiper.sendGossip(Gossiper.java:637)
	at org.apache.cassandra.gms.Gossiper.doGossipToSeed(Gossiper.java:678)
	at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:66)
	at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:178)
	at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)