[CASSANDRA-10391] sstableloader fails with client SSL enabled with non-standard keystore/truststore location

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 2.2.7, 3.0.7, 3.7
• Component/s: Legacy/Tools

Environment

[cqlsh 4.1.1 | Cassandra 2.0.14.425 | DSE 4.6.6 | CQL spec 3.1.1 | Thrift protocol 19.39.0]

[cqlsh 5.0.1 | Cassandra 2.1.8.689 | DSE 4.7.3 | CQL spec 3.2.0 | Native protocol v3]

• Severity: Normal
• Assignee: Andrew Hust
• Reporter: Jon Moses

Authors

Andrew Hust

• Created: 23/Sep/15 15:53
• Updated: 16/Apr/19 09:30
• Resolved: 19/Aug/17 18:14

Description

If client SSL is enabled, sstableloader is unable to access the keystore and truststore if they are not in the expected locations. I reproduce this issue providing -f /path/to/cassandra.yaml as well as manually using the -ks flag with the proper path to the keystore.

For example:

client_encryption_options:
    enabled: true
    keystore: /var/tmp/.keystore

# sstableloader -d 172.31.2.240,172.31.2.241 -f /etc/dse/cassandra/cassandra.yaml Keyspace1/Standard1/
Could not retrieve endpoint ranges:
java.io.FileNotFoundException: /usr/share/dse/conf/.keystore
Run with --debug to get full stack trace or --help to get help.
#
# sstableloader -d 172.31.2.240,172.31.2.241 -ks /var/tmp/.keystore Keyspace1/Standard1/
Could not retrieve endpoint ranges:
java.io.FileNotFoundException: /usr/share/dse/conf/.keystore
Run with --debug to get full stack trace or --help to get help.
#

The full stack is:

# sstableloader -d 172.31.2.240,172.31.2.241 -f /etc/dse/cassandra/cassandra.yaml --debug Keyspace1/Standard1/
Could not retrieve endpoint ranges:
java.io.FileNotFoundException: /usr/share/dse/conf/.keystore
java.lang.RuntimeException: Could not retrieve endpoint ranges:
	at org.apache.cassandra.tools.BulkLoader$ExternalClient.init(BulkLoader.java:283)
	at org.apache.cassandra.io.sstable.SSTableLoader.stream(SSTableLoader.java:144)
	at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:95)
Caused by: java.io.FileNotFoundException: /usr/share/dse/conf/.keystore
	at com.datastax.bdp.transport.client.TClientSocketFactory.getSSLSocket(TClientSocketFactory.java:128)
	at com.datastax.bdp.transport.client.TClientSocketFactory.openSocket(TClientSocketFactory.java:114)
	at com.datastax.bdp.transport.client.TDseClientTransportFactory.openTransport(TDseClientTransportFactory.java:186)
	at com.datastax.bdp.transport.client.TDseClientTransportFactory.openTransport(TDseClientTransportFactory.java:120)
	at com.datastax.bdp.transport.client.TDseClientTransportFactory.openTransport(TDseClientTransportFactory.java:111)
	at org.apache.cassandra.tools.BulkLoader$ExternalClient.createThriftClient(BulkLoader.java:302)
	at org.apache.cassandra.tools.BulkLoader$ExternalClient.init(BulkLoader.java:254)
	... 2 more
root@ip-172-31-2-240:/tmp/foo#

.

If I copy the keystore to the expected location, I get the same error with the truststore.

# sstableloader -d 172.31.2.240,172.31.2.241 -f /etc/dse/cassandra/cassandra.yaml --debug Keyspace1/Standard1/
Could not retrieve endpoint ranges:
java.io.FileNotFoundException: /usr/share/dse/conf/.truststore
java.lang.RuntimeException: Could not retrieve endpoint ranges:
	at org.apache.cassandra.tools.BulkLoader$ExternalClient.init(BulkLoader.java:283)
	at org.apache.cassandra.io.sstable.SSTableLoader.stream(SSTableLoader.java:144)
	at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:95)
Caused by: java.io.FileNotFoundException: /usr/share/dse/conf/.truststore
	at com.datastax.bdp.transport.client.TClientSocketFactory.getSSLSocket(TClientSocketFactory.java:130)
	at com.datastax.bdp.transport.client.TClientSocketFactory.openSocket(TClientSocketFactory.java:114)
	at com.datastax.bdp.transport.client.TDseClientTransportFactory.openTransport(TDseClientTransportFactory.java:186)
	at com.datastax.bdp.transport.client.TDseClientTransportFactory.openTransport(TDseClientTransportFactory.java:120)
	at com.datastax.bdp.transport.client.TDseClientTransportFactory.openTransport(TDseClientTransportFactory.java:111)
	at org.apache.cassandra.tools.BulkLoader$ExternalClient.createThriftClient(BulkLoader.java:302)
	at org.apache.cassandra.tools.BulkLoader$ExternalClient.init(BulkLoader.java:254)
	... 2 more
#

If I copy the truststore, it finds them both, but then fails to open them due to what I assume is a password error, even those it's present in the cassandra.yaml.

# sstableloader -d 172.31.2.240,172.31.2.241 -f /etc/dse/cassandra/cassandra.yaml --debug Keyspace1/Standard1/
Could not retrieve endpoint ranges:
java.io.IOException: Failed to open transport to: 172.31.2.240:9160
java.lang.RuntimeException: Could not retrieve endpoint ranges:
	at org.apache.cassandra.tools.BulkLoader$ExternalClient.init(BulkLoader.java:283)
	at org.apache.cassandra.io.sstable.SSTableLoader.stream(SSTableLoader.java:144)
	at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:95)
Caused by: java.io.IOException: Failed to open transport to: 172.31.2.240:9160
	at com.datastax.bdp.transport.client.TDseClientTransportFactory.openTransport(TDseClientTransportFactory.java:137)
	at com.datastax.bdp.transport.client.TDseClientTransportFactory.openTransport(TDseClientTransportFactory.java:111)
	at org.apache.cassandra.tools.BulkLoader$ExternalClient.createThriftClient(BulkLoader.java:302)
	at org.apache.cassandra.tools.BulkLoader$ExternalClient.init(BulkLoader.java:254)
	... 2 more
Caused by: org.apache.thrift.transport.TTransportException: Error creating the transport
	at org.apache.thrift.transport.TSSLTransportFactory.createSSLContext(TSSLTransportFactory.java:201)
	at org.apache.thrift.transport.TSSLTransportFactory.getClientSocket(TSSLTransportFactory.java:165)
	at com.datastax.bdp.transport.client.TClientSocketFactory.getSSLSocket(TClientSocketFactory.java:136)
	at com.datastax.bdp.transport.client.TClientSocketFactory.openSocket(TClientSocketFactory.java:114)
	at com.datastax.bdp.transport.client.TDseClientTransportFactory.openTransport(TDseClientTransportFactory.java:186)
	at com.datastax.bdp.transport.client.TDseClientTransportFactory.openTransport(TDseClientTransportFactory.java:120)
	... 5 more
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
	at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:772)
	at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:55)
	at java.security.KeyStore.load(KeyStore.java:1445)
	at org.apache.thrift.transport.TSSLTransportFactory.createSSLContext(TSSLTransportFactory.java:179)
	... 10 more
Caused by: java.security.UnrecoverableKeyException: Password verification failed
	at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:770)
	... 13 more

If I specify the password on the command line, I get the same error.

Issue Links

• is related to: CASSANDRA-11708 sstableloader not work with ssl options