Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-12673

Nodes cannot see each other in multi-DC, non-EC2 environment with two-interface nodes due to outbound node-to-node connection binding to private interface

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Low
    • Resolution: Fixed
    • Fix Version/s: 2.2.9, 3.0.11, 3.10
    • Component/s: Legacy/Core
    • Labels:
      None
    • Environment:

      Multi-DC, non-EC2 environment with two-interface nodes

    • Severity:
      Low

      Description

      We have a two-DC cluster in non-EC2 environment with each node containing two interfaces, one using private addresses for intra-DC communication and the other using public addresses for inter-DC communication. After proper configuration setup needed for this kind of environment we observed nodes cannot see each other.

      The configuration changes made for this purpose are as follows:

      listen_address: bound to private interface
      broadcast_address: bound to public address
      listen_on_broadcast_address: true
      endpoint_snitch: GossipingPropertyFileSnitch
      prefer_local=true (in cassandra-rackdc.properties)

      Upon restart, cassandra node contacts other nodes with their public addresses which is essential for making contacts to foreign data centers. After exhaustive investigation we found cassandra binds outbound node-to-node connections to private interface (the one specified in listen_address) that poses a problem for our environment as these data centers do not allow connections from private interface to public network.

      A portion of cassandra code responsible for local binding of outbound connections can be found in method org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket:

                  if (!Config.getOutboundBindAny())
                      channel.bind(new InetSocketAddress(FBUtilities.getLocalAddress(), 0));
      

      After we commented out these two lines and deployed cassandra.jar across the cluster, the nodes were able to see each other and everything appears to be working fine, including two-DC setup.

      Do you think it's possible to remove these two lines without negative consequences? Alternatively, if the local binding serves some specific purpose of which I'm ignorant would it be possible to make it configurable?

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                mmajercik Milan Majercik
                Reporter:
                mmajercik Milan Majercik
                Authors:
                Milan Majercik
                Reviewers:
                Paulo Motta (Deprecated)
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: