[CASSANDRA-8704] IllegalStateException when running SSTableLoader

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 2.1.3
• Component/s: Legacy/Streaming and Messaging
• Severity: Normal
• Assignee: Marcus Eriksson
• Reporter: Philip Thompson

Authors

Marcus Eriksson

Reviewers

Benedict Elliott Smith

• Created: 30/Jan/15 00:50
• Updated: 16/Apr/19 09:31
• Resolved: 30/Jan/15 11:49

Description

The sstableloader dtests are failing on 2.1-HEAD and trunk: http://cassci.datastax.com/job/cassandra-2.1_dtest/696/testReport/sstable_generation_loading_test/

The stack trace is as follows:

Streaming relevant part of /var/folders/v3/z4wf_34n1q506_xjdy49gb780000gn/T/dtest-4JCAxa/test/node1/data_copy/ks/counter1-86be6170a81911e4bb5177c771bcee1a/la-1-big-Data.db to [/127.0.0.1, /127.0.0.2]
progress: [/127.0.0.1]0:1/1 100% [/127.0.0.2]0:1/1 100% total: 100% 0  MB/s(avg: 0 MB/s)ERROR 00:47:51 BAD RELEASE: attempted to release a shared reference (org.apache.cassandra.utils.concurrent.Ref$State@47a2174b) that has already been released
ERROR 00:47:51 [Stream #9dbdade0-a819-11e4-bad4-473c1159ad6e] Streaming error occurred
java.lang.IllegalStateException: Attempted to release a reference that has already been released
	at org.apache.cassandra.utils.concurrent.Ref$State.release(Ref.java:73) ~[main/:na]
	at org.apache.cassandra.utils.concurrent.Ref.release(Ref.java:32) ~[main/:na]
	at org.apache.cassandra.streaming.StreamTransferTask.complete(StreamTransferTask.java:76) ~[main/:na]
	at org.apache.cassandra.streaming.StreamSession.received(StreamSession.java:559) ~[main/:na]
	at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:441) ~[main/:na]
	at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:252) ~[main/:na]
	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
progress: [/127.0.0.1]0:1/1 100% [/127.0.0.2]0:1/1 100% total: 100% 0  MB/s(avg: 0 MB/s)WARN  00:47:51 [Stream #9dbdade0-a819-11e4-bad4-473c1159ad6e] Stream failed
Streaming to the following hosts failed:
[/127.0.0.2]
java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed
	at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
	at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
	at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:121)
Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
	at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
	at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
	at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
	at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
	at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:209)
	at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:185)
	at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:394)
	at org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:586)
	at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:450)
	at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:252)
	at java.lang.Thread.run(Thread.java:745)

Git bisect points to #7705 as the first commit where the failures begin, though 7705 may have simply uncovered a bug, rather than caused it.

/cc krummas