Cassandra
  1. Cassandra
  2. CASSANDRA-2491

A new config parameter, broadcast_address

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Fix Version/s: 1.0.0
    • Component/s: Core
    • Labels:
      None
    • Environment:

      x86_64 GNU/Linux

      Description

      A new config parameter, broadcast_address

      In a cluster setup where one or more nodes is behind a firewall and has a private ip address, listen_address does not allow the hosts behind the firewalls to be discovered by other nodes.

      Attached is a patch that introduces a new config parameter broadcast_address which allows Cassandra nodes to explicitly specify their external ip address.
      In addition, this allows listen_address to be set to 0.0.0.0 on the already firewalled node.

      broadcast_address fallsback to listen_address when it is not stated.

      1. 2491_broadcast_address.patch
        83 kB
        Khee Chin
      2. 2491_broadcast_address_v6.patch
        85 kB
        Vijay
      3. 0001-2491-Git_Patch_v7.patch
        98 kB
        Vijay

        Issue Links

          Activity

          Hide
          Khee Chin added a comment -

          Thanks Brandon and Vijay.

          Show
          Khee Chin added a comment - Thanks Brandon and Vijay.
          Hide
          Brandon Williams added a comment -

          Committed, thanks.

          Show
          Brandon Williams added a comment - Committed, thanks.
          Hide
          Vijay added a comment -

          I am not sure why this is happening... i have tried multiple times and it worked to me... this time it is Git patch and hope this works.... i have also updated ITC with the updates to gossiper, let me know if this helps.

          Show
          Vijay added a comment - I am not sure why this is happening... i have tried multiple times and it worked to me... this time it is Git patch and hope this works.... i have also updated ITC with the updates to gossiper, let me know if this helps.
          Hide
          Brandon Williams added a comment -

          This looks good, but now I'm getting a patch error:

          patch: **** malformed patch at line 198: Index: src/java/org/apache/cassandra/db/HintedHandOffManager.java

          Show
          Brandon Williams added a comment - This looks good, but now I'm getting a patch error: patch: **** malformed patch at line 198: Index: src/java/org/apache/cassandra/db/HintedHandOffManager.java
          Hide
          Vijay added a comment - - edited

          Earlier we where not throwing the exception in v5 i was (because of the refractor)... v6 doesnt throw it...

          Show
          Vijay added a comment - - edited Earlier we where not throwing the exception in v5 i was (because of the refractor)... v6 doesnt throw it...
          Hide
          Brandon Williams added a comment -

          It goes away when the patch is removed and is 100% reproducible with the patch. All nodes are running the same version (but that shouldn't be a problem anyway.)

          Show
          Brandon Williams added a comment - It goes away when the patch is removed and is 100% reproducible with the patch. All nodes are running the same version (but that shouldn't be a problem anyway.)
          Hide
          Vijay added a comment -

          Hi Brandon,
          I doubt thats because of this patch are you running different versions on the same cluster?... i am sure that this existed in the old version too.... I can skip bytes if you want or create a different ticket to track... (http://www.datastax.com/support-forums/topic/eofexception-on-startup – Ignore the answer to the question though just looks similar).

          Show
          Vijay added a comment - Hi Brandon, I doubt thats because of this patch are you running different versions on the same cluster?... i am sure that this existed in the old version too.... I can skip bytes if you want or create a different ticket to track... ( http://www.datastax.com/support-forums/topic/eofexception-on-startup – Ignore the answer to the question though just looks similar).
          Hide
          Brandon Williams added a comment -

          I'm getting EOF exceptions on startup from ITC with this patch.

          ERROR 21:06:10,912 Fatal exception in thread Thread[Thread-6,5,main]
          java.io.IOError: java.io.EOFException
                  at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:112)
          Caused by: java.io.EOFException
                  at java.io.DataInputStream.readInt(DataInputStream.java:375)
                  at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:98)
          

          One for every node in the cluser, per machine.

          Show
          Brandon Williams added a comment - I'm getting EOF exceptions on startup from ITC with this patch. ERROR 21:06:10,912 Fatal exception in thread Thread[Thread-6,5,main] java.io.IOError: java.io.EOFException at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:112) Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:375) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:98) One for every node in the cluser, per machine.
          Hide
          Vijay added a comment -

          Sure... will try that in another ticket... Thanks...

          Show
          Vijay added a comment - Sure... will try that in another ticket... Thanks...
          Hide
          Brandon Williams added a comment -

          One last thing I overlooked earlier is the snitches have getBCA(). This seems unnecessary since anything that needs it can access DD, which is all the snitches are doing. This changes the snitch interface and breaks any custom snitches.

          Show
          Brandon Williams added a comment - One last thing I overlooked earlier is the snitches have getBCA(). This seems unnecessary since anything that needs it can access DD, which is all the snitches are doing. This changes the snitch interface and breaks any custom snitches.
          Hide
          Vijay added a comment -

          Thanks Brandon, Plz find the attached.

          Show
          Vijay added a comment - Thanks Brandon, Plz find the attached.
          Hide
          Brandon Williams added a comment -

          That makes sense, as long as it's doing it before gossip starts like the ec2snitch currently does for dc/rack.

          Show
          Brandon Williams added a comment - That makes sense, as long as it's doing it before gossip starts like the ec2snitch currently does for dc/rack.
          Hide
          Vijay added a comment -

          Hi Brandon,

          "Can you explain why that's necessary? It seems to me if the node doesn't know it's own BA when it starts up, it's already going to get into trouble."

          – This is Just for automation, Snitch will be initialized when DatabaseDescripter is initialized hence it will be safe and the Snitch is called only for the first time to set the BA... The fall-back if BA is not available then use getLocalAddress()
          – It Can also be automated outside cassandra too... let me know if you still think it is bad i can remove that logic.

          For the rest, I will rebase it once again if you say we are good to go...

          Show
          Vijay added a comment - Hi Brandon, "Can you explain why that's necessary? It seems to me if the node doesn't know it's own BA when it starts up, it's already going to get into trouble." – This is Just for automation, Snitch will be initialized when DatabaseDescripter is initialized hence it will be safe and the Snitch is called only for the first time to set the BA... The fall-back if BA is not available then use getLocalAddress() – It Can also be automated outside cassandra too... let me know if you still think it is bad i can remove that logic. For the rest, I will rebase it once again if you say we are good to go...
          Hide
          Brandon Williams added a comment - - edited

          Mostly looks good. A few things I noticed:

          • There's a typo in "receiveMessage"
          • SimpleSnitch and AbstractNetworkTopologySnitch are importing DD but not using it
          • the VersionedValue addition probably doesn't belong here

          Allow brodcastAddress to be set by snitch so we can automate via snitch (property or EC2MultiregionSnitch).

          Can you explain why that's necessary? It seems to me if the node doesn't know it's own BA when it starts up, it's already going to get into trouble.

          Show
          Brandon Williams added a comment - - edited Mostly looks good. A few things I noticed: There's a typo in "receiveMessage" SimpleSnitch and AbstractNetworkTopologySnitch are importing DD but not using it the VersionedValue addition probably doesn't belong here Allow brodcastAddress to be set by snitch so we can automate via snitch (property or EC2MultiregionSnitch). Can you explain why that's necessary? It seems to me if the node doesn't know it's own BA when it starts up, it's already going to get into trouble.
          Hide
          Vijay added a comment -

          Rebased again....

          Show
          Vijay added a comment - Rebased again....
          Hide
          Vijay added a comment -

          Ooops.... was not intentional (except changing it to getBCA() because of the rebase.... svn patch created a mess... i will rebase it again...

          Show
          Vijay added a comment - Ooops.... was not intentional (except changing it to getBCA() because of the rebase.... svn patch created a mess... i will rebase it again...
          Hide
          Brandon Williams added a comment -

          Why the changes to StreamRequestMessage?

          Show
          Brandon Williams added a comment - Why the changes to StreamRequestMessage?
          Hide
          Vijay added a comment -

          1) Rebased
          2) Attached patch has modification to IncomingTcpConnection (re factor) in order to support Gossiper.instance.setVersion as versions should be on broadcast IP and socket.getInetAddress will break.
          3) Allow brodcastAddress to be set by snitch so we can automate via snitch (property or EC2MultiregionSnitch).

          Show
          Vijay added a comment - 1) Rebased 2) Attached patch has modification to IncomingTcpConnection (re factor) in order to support Gossiper.instance.setVersion as versions should be on broadcast IP and socket.getInetAddress will break. 3) Allow brodcastAddress to be set by snitch so we can automate via snitch (property or EC2MultiregionSnitch).
          Hide
          Khee Chin added a comment -

          In our present situation, in a varsity lan, (where the ips are 10.0.x.y) some of our servers are behind an additional router (the router has a 10.0.x.y address, while its clients have 192.168.x.y), and are not accessible by the nodes which happen to have a 10.0.x.y ip address.

          In a simplified example (with four nodes)
          A - 10.0.0.1
          B - 10.0.0.2
          C - 192.168.0.1 (behind router with 10.0.0.3)
          D - 192.168.0.2 (behind router with 10.0.0.4)

          Thus, we can't set C or D to listen on the external ip (10.0.0.3|4) since they can't bind to it. With the attached patch, they can set listen_address to 0.0.0.0 or 192.168.0.1|2 and broadcast_address to 10.0.0.3|4 .

          This is a non-typical scenario, and not many admins will need to configure this, thus, broadcast_address fallsback to listen_address when it is not set.

          Thanks.

          Show
          Khee Chin added a comment - In our present situation, in a varsity lan, (where the ips are 10.0.x.y) some of our servers are behind an additional router (the router has a 10.0.x.y address, while its clients have 192.168.x.y), and are not accessible by the nodes which happen to have a 10.0.x.y ip address. In a simplified example (with four nodes) A - 10.0.0.1 B - 10.0.0.2 C - 192.168.0.1 (behind router with 10.0.0.3) D - 192.168.0.2 (behind router with 10.0.0.4) Thus, we can't set C or D to listen on the external ip (10.0.0.3|4) since they can't bind to it. With the attached patch, they can set listen_address to 0.0.0.0 or 192.168.0.1|2 and broadcast_address to 10.0.0.3|4 . This is a non-typical scenario, and not many admins will need to configure this, thus, broadcast_address fallsback to listen_address when it is not set. Thanks.
          Hide
          Jonathan Ellis added a comment -

          why wouldn't you just set listen_address=external ip?

          Show
          Jonathan Ellis added a comment - why wouldn't you just set listen_address=external ip?
          Hide
          Florent Clairambault added a comment -

          I think a lot of people are interested in having this feature. If you could apply this patch to the cassandra trunk that would be great!

          Show
          Florent Clairambault added a comment - I think a lot of people are interested in having this feature. If you could apply this patch to the cassandra trunk that would be great!

            People

            • Assignee:
              Vijay
              Reporter:
              Khee Chin
              Reviewer:
              Brandon Williams
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 336h
                336h
                Remaining:
                Remaining Estimate - 336h
                336h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development