[CASSANDRA-11583] Exception when streaming sstables using `sstableloader` - ASF JIRA

Details

Type: Bug
Status: Open
Priority: Normal
Resolution: Unresolved
Fix Version/s: None
Component/s: Legacy/Tools
Labels:
None
Environment:

$ uname -a
Linux bigdb-100 3.2.0-99-virtual #139-Ubuntu SMP Mon Feb 1 23:52:21 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
I am using Datastax Enterprise 4.7.8-1 which is based on 2.1.13.

Severity:
Normal

Description

This bug came out of ~~CASSANDRA-11562~~.

I have have a keyspace snapshotted from a 2.1.11 (DSE 4.7.5-1) node. When I'm running the `sstableloader` I get the following output/exception:

# sstableloader --nodes X.X.X.20 --username YYY --password ZZZ --ignore XXX /var/lib/cassandra/data/XXX/ZZZ-f7ebdf0daa3a3062828fddebc109a3b2
Established connection to initial hosts
Opening sstables and calculating sections to stream
Streaming relevant part of /var/lib/cassandra/data/XXX/ZZZ-f7ebdf0daa3a3062828fddebc109a3b2/XXX-ZZZ-ka-6463-Data.db /var/lib/cassandra/data/XXX/ZZZ-f7ebdf0daa3a3062828fddebc109a3b2/tink-ZZZ-ka-6464-Data.db to [/X.X.X.33, /X.X.X.113, /X.X.X.32, /X.X.X.20, /X.X.X.122, /X.X.X.176, /X.X.X.143, /X.X.X.172, /X.X.X.50, /X.X.X.51, /X.X.X.52, /X.X.X.71, /X.X.X.53, /X.X.X.54, /X.X.X.47, /X.X.X.31, /X.X.X.8]
progress: [/X.X.X.113]0:0/2 0  % [/X.X.X.143]0:0/2 0  % [/X.X.X.172]0:0/2 0  % [/X.X.X.20]0:0/2 0  % [/X.X.X.71]0:0/2 0  % [/X.X.X.122]0:0/2 0  % [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:0/2 0  % [/X.X.X.143]0:0/2 0  % [/X.X.X.172]0:0/2 0  % [/X.X.X.20]0:1/2 1  % [/X.X.X.71]0:0/2 0  % [/X.X.X.122]0:0/2 0  % [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:0/2 0  % [/X.X.X.143]0:0/2 1  % [/X.X.X.172]0:0/2 0  % [/X.X.X.20]0:1/2 1  % [/X.X.X.71]0:0/2 0  % [/X.X.X.122]0:0/2 0  % [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:0/2 0  % [/X.X.X.143]0:1/2 1  % [/X.X.X.172]0:0/2 0  % [/X.X.X.20]0:1/2 1  % [/X.X.X.71]0:0/2 0  % [/X.X.X.122]0:0/2 0  % [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:0/2 0  % [/X.X.X.143]0:1/2 1  % [/X.X.X.172]0:0/2 0  % [/X.X.X.20]0:1/2 1  % [/X.X.X.71]0:1/2 1  % [/X.X.X.122]0:0/2 0  % [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:0/2 0  % [/X.X.X.143]0:1/2 1  % [/X.X.X.172]0:0/2 0  % [/X.X.X.20]0:1/2 1  % [/X.X.X.71]0:1/2 1  % [/X.X.X.122]0:1/2 1  % [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:0/2 0  % [/X.X.X.143]0:1/2 1  % [/X.X.X.172]0:0/2 0  % [/X.X.X.20]0:1/2 1  % [/X.X.X.71]0:1/2 1  % [/X.X.X.122]0:1/2 1  % [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:0/2 7  % [/X.X.X.143]0:1/2 1  % [/X.X.X.172]0:0/2 0  % [/X.X.X.20]0:1/2 1  % [/X.X.X.71]0:1/2 1  % [/X.X.X.122]0:1/2 1  % [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:0/2 7  % [/X.X.X.143]0:1/2 6  % [/X.X.X.172]0:0/2 0  % [/X.X.X.20]0:1/2 1  % [/X.X.X.71]0:1/2 1  % [/X.X.X.122]0:1/2 1  % [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:0/2 12 % [/X.X.X.143]0:1/2 6  % [/X.X.X.172]0:0/2 0  % [/X.X.X.20]0:1/2 1  % [/X.X.X.71]0:1/2 1  % [/X.X.X.122]0:1/2 1  % [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:0/2 12 % [/X.X.X.143]0:1/2 11 % [/X.X.X.172]0:0/2 0  % [/X.X.X.20]0:1/2 1  % [/X.X.X.71]0:1/2 1  % [/X.X.X.122]0:1/2 1  % [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:0/2 19 % [/X.X.X.143]0:1/2 11 % [/X.X.X.172]0:0/2 0  % [/X.X.X.20]0:1/2 1  % [/X.X.X.71]0:1/2 1  % [/X.X.X.122]0:1/2 1  % [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:0/2 19 % [/X.X.X.143]0:1/2 15 % [/X.X.X.172]0:0/2 0  % [/X.X.X.20]0:1/2 1  % [/X.X.X.71]0:1/2 1  % [/X.X.X.122]0:1/2 1  % [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:0/2 26 % [/X.X.X.143]0:1/2 15 % [/X.X.X.172]0:0/2 0  % [/X.X.X.20]0:1/2 1  % [/X.X.X.71]0:1/2 1  % [/X.X.X.122]0:1/2 1  % [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:0/2 26 % [/X.X.X.143]0:1/2 20 % [/X.X.X.172]0:0/2 0  % [/X.X.X.20]0:1/2 1  % [/X.X.X.71]0:1/2 1  % [/X.X.X.122]0:1/2 1  % [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:0/2 26 % [/X.X.X.143]0:1/2 21 % [/X.X.X.172]0:0/2 0  % [/X.X.X.20]0:1/2 1  % [/X.X.X.71]0:1/2 1  % [/X.X.X.122]0:1/2 1  % [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:0/2 26 % [/X.X.X.143]0:1/2 21 % [/X.X.X.172]0:0/2 0  % [/X.X.X.20]0:1/2 3  % [/X.X.X.71]0:1/2 1  % [/X.X.X.122]0:1/2 1  % [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:0/2 42 % [/X.X.X.143]0:1/2 27 % [/X.X.X.172]0:0/2 0  % [/X.X.X.20]0:1/2 3  % [/X.X.X.71]0:1/2 6  % [/X.X.X.122]0:1/2 1  % [/X.X.X.47]0:0/2 
[...]
progress: [/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100% [/X.X.X.172]0:0/2 78 % [/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100% [/X.X.X.122]0:1/2 97 % [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100% [/X.X.X.172]0:0/2 78 % [/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100% [/X.X.X.122]0:1/2 97 % [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100% [/X.X.X.172]0:0/2 86 % [/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100% [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100% [/X.X.X.172]0:0/2 86 % [/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100% [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100% [/X.X.X.172]0:0/2 86 % [/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100% [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100% [/X.X.X.172]0:0/2 86 % [/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100% [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100% [/X.X.X.172]0:0/2 86 % [/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100% [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100% [/X.X.X.172]0:0/2 95 % [/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100% [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100% [/X.X.X.172]0:0/2 95 % [/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100% [/X.X.X.47]0:1/2 progress: [/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100% [/X.X.X.172]0:0/2 95 % [/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100% [/X.X.X.47]0:2/2 progress: [/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100% [/X.X.X.172]0:0/2 95 % [/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100% [/X.X.X.47]0:2/2 progress: [/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100% [/X.X.X.172]0:1/2 98 % [/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100% [/X.X.X.47]0:2/2 progress: [/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100% [/X.X.X.172]0:1/2 99 % [/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100% [/X.X.X.47]0:2/2 progress: [/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100% [/X.X.X.172]0:2/2 100% [/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100% [/X.X.X.47]0:2/2 progress: [/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100% [/X.X.X.172]0:2/2 100% [/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100% [/X.X.X.47]0:2/2 progress: [/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100% [/X.X.X.172]0:2/2 100% [/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100% [/X.X.X.47]0:2/2 progress: [/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100% [/X.X.X.172]0:2/2 100% [/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100% [/X.X.X.47]0:2/2 progress: [/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100% [/X.X.X.172]0:2/2 100% [/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100% [/X.X.X.47]0:2/2 progress: [/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100% [/X.X.X.172]0:2/2 100% [/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100% [/X.X.X.47]0:2/2 progress: [/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100% [/X.X.X.172]0:2/2 100% [/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100% [/X.X.X.47]0:2/2 progress: [/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100% [/X.X.X.172]0:2/2 100% [/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100% [/X.X.X.47]0:2/2 progress: [/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100% [/X.X.X.172]0:2/2 100% [/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100% [/X.X.X.47]0:2/2 progress: [/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100% [/X.X.X.172]0:2/2 100% [/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100% [/X.X.X.47]0:2/2 100% [/X.X.X.176]0:2/2 100% [/X.X.X.8]0:2/2 100% total: 100% 0  MB/s(avg: 23 MB/s)
Summary statistics:
   Connections per host:         : 1
   Total files transferred:      : 14
   Total bytes transferred:      : 142070241
   Total duration (ms):          : 5823
   Average transfer rate (MB/s): : 23
   Peak transfer rate (MB/s):    : 33

ERROR 09:07:12 [Stream #6e345de0-02e9-11e6-b944-1fee10beb91b] Streaming error occurred
java.lang.AssertionError: null
        at org.apache.cassandra.io.sstable.SSTableLoader.releaseReferences(SSTableLoader.java:208) ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218]
        at org.apache.cassandra.io.sstable.SSTableLoader.onSuccess(SSTableLoader.java:193) ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218]
        at org.apache.cassandra.io.sstable.SSTableLoader.onSuccess(SSTableLoader.java:48) ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218]
        at com.google.common.util.concurrent.Futures$4.run(Futures.java:1181) ~[guava-16.0.1.jar:na]
        at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.1.jar:na]
        at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-16.0.1.jar:na]
        at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-16.0.1.jar:na]
        at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:185) ~[guava-16.0.1.jar:na]
        at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:213) ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218]
        at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:184) ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218]
        at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:415) ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218]
        at org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:607) ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218]
        at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:471) ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218]
        at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:263) ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]

Question: Is it an issue that I dumped the sstables using a slightly older Cassandra node (2.1.11 instead of 2.1.13)? Is streaming from 2.1.13 to 2.1.11 incompatible?

Let me know if you need anything else from me. Also, know of any possible for workaround?

Attachments

Issue Links

is related to

CASSANDRA-11562 "Could not retrieve endpoint ranges" for sstableloader

Resolved

Exception when streaming sstables using `sstableloader`

Details

Description

Attachments

Issue Links

Activity

People

Dates