Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-14437

SSTableLoader does not work when "internode_encryption : all" is set

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Normal
    • Resolution: Unresolved
    • Fix Version/s: 3.11.x
    • Component/s: Legacy/Tools
    • Labels:
      None
    • Severity:
      Normal

      Description

      I am trying to use sstableloader to restore snapshot.

      If "internode_encryption :  all" is set, then it does not work and complain with below error messages. I initiated sstableloader from 10.1.10.203 (yvr-paul-cas003), so 10.1.10.203 worked fine, but the the other two nodes (10.1.10.201 & 10.1.10.202 failed)

      pcheon@yvr-paul-cas003:~/t$ sstableloader -v -d 10.1.10.203 office_audit/log/ -f /etc/cassandra/cassandra.yaml -u pcheon -pw `cat .secret`
      WARN  17:23:45,166 Small commitlog volume detected at /var/lib/cassandra/commitlog; setting commitlog_total_space_in_mb to 2316.  You can override this in cassandra.yaml
      WARN  17:23:45,170 Small cdc volume detected at /var/lib/cassandra/cdc_raw; setting cdc_total_space_in_mb to 1158.  You can override this in cassandra.yaml
      WARN  17:23:45,285 Only 5.318GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots
      Established connection to initial hosts
      Opening sstables and calculating sections to stream
      Streaming relevant part of /home/pcheon/t/office_audit/log/mc-1083-big-Data.db /home/pcheon/t/office_audit/log/mc-1100-big-Data.db /home/pcheon/t/office_audit/log/mc-1101-big-Data.db /home/pcheon/t/office_audit/log/mc-257-big-Data.db /home/pcheon/t/office_audit/log/mc-984-big-Data.db  to [/10.1.10.201, /10.1.10.203, /10.1.10.202]
      ERROR 17:23:49,460 [Stream #938baee0-4e2d-11e8-9be0-ebc69ba4b87f] Streaming error occurred on session with peer 10.1.10.201
      java.net.SocketException: Invalid argument or cannot assign requested address
      	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_112]
      	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_112]
      	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_112]
      	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_112]
      	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_112]
      	at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_112]
      	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668) ~[na:1.8.0_112]
      	at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:495) ~[na:1.8.0_112]
      	at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:169) ~[na:1.8.0_112]
      	at org.apache.cassandra.security.SSLFactory.getSocket(SSLFactory.java:81) ~[apache-cassandra-3.11.2.jar:3.11.2]
      	at org.apache.cassandra.tools.BulkLoadConnectionFactory.createConnection(BulkLoadConnectionFactory.java:56) ~[apache-cassandra-3.11.2.jar:3.11.2]
      	at org.apache.cassandra.streaming.StreamSession.createConnection(StreamSession.java:282) ~[apache-cassandra-3.11.2.jar:3.11.2]
      	at org.apache.cassandra.streaming.ConnectionHandler.initiate(ConnectionHandler.java:86) ~[apache-cassandra-3.11.2.jar:3.11.2]
      	at org.apache.cassandra.streaming.StreamSession.start(StreamSession.java:269) ~[apache-cassandra-3.11.2.jar:3.11.2]
      	at org.apache.cassandra.streaming.StreamCoordinator$StreamSessionConnector.run(StreamCoordinator.java:263) [apache-cassandra-3.11.2.jar:3.11.2]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_112]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_112]
      	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.2.jar:3.11.2]
      	at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_112]
      ERROR 17:23:49,458 [Stream #938baee0-4e2d-11e8-9be0-ebc69ba4b87f] Streaming error occurred on session with peer 10.1.10.202
      java.net.SocketException: Invalid argument or cannot assign requested address
      	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_112]
      	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_112]
      	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_112]
      	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_112]
      	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_112]
      	at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_112]
      	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668) ~[na:1.8.0_112]
      	at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:495) ~[na:1.8.0_112]
      	at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:169) ~[na:1.8.0_112]
      	at org.apache.cassandra.security.SSLFactory.getSocket(SSLFactory.java:81) ~[apache-cassandra-3.11.2.jar:3.11.2]
      	at org.apache.cassandra.tools.BulkLoadConnectionFactory.createConnection(BulkLoadConnectionFactory.java:56) ~[apache-cassandra-3.11.2.jar:3.11.2]
      	at org.apache.cassandra.streaming.StreamSession.createConnection(StreamSession.java:282) ~[apache-cassandra-3.11.2.jar:3.11.2]
      	at org.apache.cassandra.streaming.ConnectionHandler.initiate(ConnectionHandler.java:86) ~[apache-cassandra-3.11.2.jar:3.11.2]
      	at org.apache.cassandra.streaming.StreamSession.start(StreamSession.java:269) ~[apache-cassandra-3.11.2.jar:3.11.2]
      	at org.apache.cassandra.streaming.StreamCoordinator$StreamSessionConnector.run(StreamCoordinator.java:263) [apache-cassandra-3.11.2.jar:3.11.2]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_112]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_112]
      	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.2.jar:3.11.2]
      	at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_112]
      progress: total: 100% 0.000KiB/s (avg: 0.000KiB/s)
      progress: total: 100% 0.000KiB/s (avg: 0.000KiB/s)
      progress: [/10.1.10.203]0:0/5 7  % total: 7% 13.858MiB/s (avg: 2.147MiB/s)
      progress: [/10.1.10.203]0:0/5 15 % total: 15% 41.587MiB/s (avg: 4.083MiB/s)
      progress: [/10.1.10.203]0:0/5 23 % total: 23% 75.631MiB/s (avg: 5.964MiB/s)
      progress: [/10.1.10.203]0:0/5 31 % total: 31% 34.837MiB/s (avg: 7.523MiB/s)
      progress: [/10.1.10.203]0:0/5 39 % total: 39% 21.321MiB/s (avg: 8.641MiB/s)
      progress: [/10.1.10.203]0:1/5 41 % total: 41% 17.665MiB/s (avg: 8.766MiB/s)
      progress: [/10.1.10.203]0:1/5 49 % total: 49% 17.841MiB/s (avg: 9.557MiB/s)
      progress: [/10.1.10.203]0:1/5 56 % total: 56% 75.654MiB/s (avg: 10.889MiB/s)
      progress: [/10.1.10.203]0:1/5 64 % total: 64% 21.252MiB/s (avg: 11.583MiB/s)
      progress: [/10.1.10.203]0:1/5 72 % total: 72% 33.267MiB/s (avg: 12.472MiB/s)
      progress: [/10.1.10.203]0:1/5 80 % total: 80% 22.969MiB/s (avg: 13.060MiB/s)
      progress: [/10.1.10.203]0:2/5 81 % total: 81% 6.191MiB/s (avg: 12.951MiB/s)
      progress: [/10.1.10.203]0:3/5 81 % total: 81% 100.225MiB/s (avg: 12.994MiB/s)
      progress: [/10.1.10.203]0:3/5 89 % total: 89% 12.939MiB/s (avg: 12.989MiB/s)
      progress: [/10.1.10.203]0:3/5 97 % total: 97% 65.803MiB/s (avg: 13.899MiB/s)
      progress: [/10.1.10.203]0:4/5 99 % total: 99% 69.864MiB/s (avg: 14.038MiB/s)
      progress: [/10.1.10.203]0:5/5 100% total: 100% 56.930MiB/s (avg: 14.141MiB/s)
      progress: [/10.1.10.203]0:5/5 100% total: 100% 0.000KiB/s (avg: 13.373MiB/s)
      WARN  17:23:54,923 [Stream #938baee0-4e2d-11e8-9be0-ebc69ba4b87f] Stream failed
      Streaming to the following hosts failed:
      [/10.1.10.201, /10.1.10.202]
      java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed
      	at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
      	at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
      	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
      	at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:98)
      	at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:48)
      Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
      	at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:88)
      	at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
      	at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
      	at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
      	at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
      	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
      	at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:215)
      	at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:191)
      	at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:481)
      	at org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:682)
      	at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:532)
      	at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:317)
      	at java.lang.Thread.run(Thread.java:745)
      Exception in thread "main" org.apache.cassandra.tools.BulkLoadException: java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed
      	at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:114)
      	at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:48)
      Caused by: java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed
      	at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
      	at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
      	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
      	at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:98)
      	... 1 more
      Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
      	at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:88)
      	at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
      	at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
      	at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
      	at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
      	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
      	at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:215)
      	at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:191)
      	at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:481)
      	at org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:682)
      	at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:532)
      	at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:317)
      	at java.lang.Thread.run(Thread.java:745)
      

      However, if I set the "internode_encryption : none", then it works fine as expected

      I do not know why it does not work. I guess somehow sstableloader does not know how to talk with other Cassandra through encrypted channel?

      Would you help?

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              paul.cheon@visiercorp.com Paul Cheon
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: