[CASSANDRA-14522] sstableloader should use discovered broadcast address to connect intra-cluster

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.18, 3.11.4
• Component/s: Legacy/Tools
• Labels: lhf
• Severity: Normal
• Assignee: Jeremy
• Reporter: Jeremy Hanna

Authors

Jeremy

• Created: 14/Jun/18 18:47
• Updated: 16/Apr/19 09:29
• Resolved: 06/Aug/18 16:12

Description

Currently, in the LoaderOptions for the BulkLoader, the user can give a list of initial host addresses. That's to do the initial connection to the cluster but also to stream the sstables. If you have two physical interfaces, one for rpc, the other for internode traffic, then bulk loader won't currently work. It will throw an error such as:

> sstableloader -v -u cassadmin -pw xxx -d 10.133.210.101,10.133.210.102,10.133.210.103,10.133.210.104 /var/lib/cassandra/commitlog/backup_tmp/test_bkup/bkup_tbl
Established connection to initial hosts
Opening sstables and calculating sections to stream
Streaming relevant part of /var/lib/cassandra/commitlog/backup_tmp/test_bkup/bkup_tbl/mc-1-big-Data.db /var/lib/cassandra/commitlog/backup_tmp/test_bkup/bkup_tbl/mc-2-big-Data.db to [/10.133.210.101, /10.133.210.103, /10.133.210.102, /10.133.210.104]
progress: total: 100% 0 MB/s(avg: 0 MB/s)ERROR 10:16:05,311 Stream #9ed00130-6ff6-11e8-965c-93a78bf96e60 Streaming error occurred
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method) ~[na:1.8.0_101]
at sun.nio.ch.Net.connect(Net.java:454) ~[na:1.8.0_101]
at sun.nio.ch.Net.connect(Net.java:446) ~[na:1.8.0_101]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[na:1.8.0_101]
at java.nio.channels.SocketChannel.open(SocketChannel.java:189) ~[na:1.8.0_101]
at org.apache.cassandra.tools.BulkLoadConnectionFactory.createConnection(BulkLoadConnectionFactory.java:60) ~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
at org.apache.cassandra.streaming.StreamSession.createConnection(StreamSession.java:266) ~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
at org.apache.cassandra.streaming.ConnectionHandler.initiate(ConnectionHandler.java:86) ~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
at org.apache.cassandra.streaming.StreamSession.start(StreamSession.java:253) ~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
at org.apache.cassandra.streaming.StreamCoordinator$StreamSessionConnector.run(StreamCoordinator.java:212) [cassandra-all-3.0.15.2128.jar:3.0.15.2128]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_101]
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) [cassandra-all-3.0.15.2128.jar:3.0.15.2128]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-all-4.0.54.Final.jar:4.0.54.Final]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_101]
ERROR 10:16:05,312 Stream #9ed00130-6ff6-11e8-965c-93a78bf96e60 Streaming error occurred
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method) ~[na:1.8.0_101]
at sun.nio.ch.Net.connect(Net.java:454) ~[na:1.8.0_101]
at sun.nio.ch.Net.connect(Net.java:446) ~[na:1.8.0_101]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[na:1.8.0_101]
at java.nio.channels.SocketChannel.open(SocketChannel.java:189) ~[na:1.8.0_101]
at org.apache.cassandra.tools.BulkLoadConnectionFactory.createConnection(BulkLoadConnectionFactory.java:60) ~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
at org.apache.cassandra.streaming.StreamSession.createConnection(StreamSession.java:266) ~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
at org.apache.cassandra.streaming.ConnectionHandler.initiate(ConnectionHandler.java:86) ~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
at org.apache.cassandra.streaming.StreamSession.start(StreamSession.java:253) ~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
at org.apache.cassandra.streaming.StreamCoordinator$StreamSessionConnector.run(StreamCoordinator.java:212) [cassandra-all-3.0.15.2128.jar:3.0.15.2128]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_101]
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) [cassandra-all-3.0.15.2128.jar:3.0.15.2128]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-all-4.0.54.Final.jar:4.0.54.Final]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_101]
ERROR 10:16:05,312 Stream #9ed00130-6ff6-11e8-965c-93a78bf96e60 Streaming error occurred
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method) ~[na:1.8.0_101]
at sun.nio.ch.Net.connect(Net.java:454) ~[na:1.8.0_101]
at sun.nio.ch.Net.connect(Net.java:446) ~[na:1.8.0_101]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[na:1.8.0_101]
at java.nio.channels.SocketChannel.open(SocketChannel.java:189) ~[na:1.8.0_101]
at org.apache.cassandra.tools.BulkLoadConnectionFactory.createConnection(BulkLoadConnectionFactory.java:60) ~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
at org.apache.cassandra.streaming.StreamSession.createConnection(StreamSession.java:266) ~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
at org.apache.cassandra.streaming.ConnectionHandler.initiate(ConnectionHandler.java:86) ~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
at org.apache.cassandra.streaming.StreamSession.start(StreamSession.java:253) ~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
at org.apache.cassandra.streaming.StreamCoordinator$StreamSessionConnector.run(StreamCoordinator.java:212) [cassandra-all-3.0.15.2128.jar:3.0.15.2128]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_101]
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) [cassandra-all-3.0.15.2128.jar:3.0.15.2128]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-all-4.0.54.Final.jar:4.0.54.Final]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_101]
progress: total: 100% 0 MB/s(avg: 0 MB/s)WARN 10:16:05,320 Stream #9ed00130-6ff6-11e8-965c-93a78bf96e60 Stream failed
Streaming to the following hosts failed:
[/10.133.210.101, /10.133.210.103, /10.133.210.102]
java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed
at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:122)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.datastax.bdp.tools.ShellToolWrapper.main(ShellToolWrapper.java:34)
Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:215)
at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:191)
at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:449)
at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:549)
at org.apache.cassandra.streaming.StreamSession.start(StreamSession.java:259)
at org.apache.cassandra.streaming.StreamCoordinator$StreamSessionConnector.run(StreamCoordinator.java:212)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:745)
WARN 10:16:05,322 Stream #9ed00130-6ff6-11e8-965c-93a78bf96e60 Stream failed
WARN 10:16:05,322 Stream #9ed00130-6ff6-11e8-965c-93a78bf96e60 Stream failed

Issue Links

• is duplicated by: CASSANDRA-13639 SSTableLoader always uses hostname to stream files from
• is related to: CASSANDRA-14623 sstableloader should use discovered broadcast address to connect intra-cluster in 2.2