[CASSANDRA-11562] "Could not retrieve endpoint ranges" for sstableloader

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Duplicate
• Component/s: Legacy/Tools

Environment

$ uname -a
Linux bigdb-100 3.2.0-99-virtual #139-Ubuntu SMP Mon Feb 1 23:52:21 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

I am using Datastax Enterprise 4.7.5-1 which is based on 2.1.11.

• Severity: Normal
• Assignee: Unassigned
• Reporter: Jens Rantil
• Created: 13/Apr/16 15:26
• Updated: 16/Apr/19 09:30
• Resolved: 15/Apr/16 09:25

Description

I am setting up a second datacenter and have a very slow and shaky VPN connection to my old datacenter. To speed up import process I am trying to seed the new datacenter with a backup (that has been transferred encrypted out of bands from the VPN). When this is done I will issue a final clusterwide repair.

However...sstableloader crashes with the following:

sstableloader -v --nodes XXX --username MYUSERNAME --password MYPASSWORD --ignore YYY,ZZZ ./backupdir/MYKEYSPACE/MYTABLE/
Could not retrieve endpoint ranges:
java.lang.IllegalArgumentException
java.lang.RuntimeException: Could not retrieve endpoint ranges:
        at org.apache.cassandra.tools.BulkLoader$ExternalClient.init(BulkLoader.java:338)
        at org.apache.cassandra.io.sstable.SSTableLoader.stream(SSTableLoader.java:156)
        at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:106)
Caused by: java.lang.IllegalArgumentException
        at java.nio.Buffer.limit(Buffer.java:267)
        at org.apache.cassandra.utils.ByteBufferUtil.readBytes(ByteBufferUtil.java:543)
        at org.apache.cassandra.serializers.CollectionSerializer.readValue(CollectionSerializer.java:124)
        at org.apache.cassandra.serializers.MapSerializer.deserializeForNativeProtocol(MapSerializer.java:101)
        at org.apache.cassandra.serializers.MapSerializer.deserializeForNativeProtocol(MapSerializer.java:30)
        at org.apache.cassandra.serializers.CollectionSerializer.deserialize(CollectionSerializer.java:50)
        at org.apache.cassandra.db.marshal.AbstractType.compose(AbstractType.java:68)
        at org.apache.cassandra.cql3.UntypedResultSet$Row.getMap(UntypedResultSet.java:287)
        at org.apache.cassandra.config.CFMetaData.fromSchemaNoTriggers(CFMetaData.java:1833)
        at org.apache.cassandra.config.CFMetaData.fromThriftCqlRow(CFMetaData.java:1126)
        at org.apache.cassandra.tools.BulkLoader$ExternalClient.init(BulkLoader.java:330)
        ... 2 more

(where YYY,ZZZ are nodes in the old DC)

The files in ./backupdir/MYKEYSPACE/MYTABLE/ are an exact copy of a snapshot from the older datacenter that has been taken with the exact same version of Datastax Enterprise/Cassandra. The backup was taken 2-3 days ago.

Question: ./backupdir/MYKEYSPACE/MYTABLE/ contains the non-"*.db" file "manifest.json". Is that an issue?

My workaround for my quest will probably be to copy the snapshot directories out to the nodes of the new datacenter and do a DC-local repair+cleanup.

Let me know if I can assist in debugging this further.

References:

• This might be a duplicate of https://issues.apache.org/jira/browse/CASSANDRA-10629.
• http://stackoverflow.com/q/34757922/260805. http://stackoverflow.com/a/35213418/260805 claims this could happen when dropping a column, but don't think I've dropped any column for this column ever.
• http://stackoverflow.com/q/28632555/260805
• http://stackoverflow.com/q/34487567/260805

Issue Links

• duplicates: CASSANDRA-10700 2.1 sstableloader will fail if there are collections in the schema tables
• relates to: CASSANDRA-11583 Exception when streaming sstables using `sstableloader`