Cassandra
  1. Cassandra
  2. CASSANDRA-3590

Use multiple connection to share the OutboutTCPConnection

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: Core
    • Labels:
      None

      Description

      Currently there is one connection between any given host to another host in the cluster, the problem with this is:
      1) This can become a bottleneck in some cases where the latencies are higher.
      2) When a connection is dropped we also drop the queue and recreate a new one and hence the messages can be lost (Currently hints will take care of it and clients also can retry)

      by making it a configurable option to configure the number of connections and also making the queue common to those connections the above 2 issues can be resolved.

      1. TCPTest.xlsx
        205 kB
        Vijay
      2. TCPTest.zip
        8 kB
        Vijay

        Activity

        Hide
        Brandon Williams added a comment -

        1) This can become a bottleneck in some cases where the latencies are higher.

        I think we should see how CASSANDRA-3378 works out before jumping to this level of complexity.

        by making it a configurable option to configure the number of connections and also making the queue common to those connections

        I think this might get hairy with regard to protocol versioning, since OTC is where it is always set.

        Show
        Brandon Williams added a comment - 1) This can become a bottleneck in some cases where the latencies are higher. I think we should see how CASSANDRA-3378 works out before jumping to this level of complexity. by making it a configurable option to configure the number of connections and also making the queue common to those connections I think this might get hairy with regard to protocol versioning, since OTC is where it is always set.
        Hide
        Jonathan Ellis added a comment -

        Not sure about this one:

        1. Adding another TCP stream shouldn't give you more throughput or lower latency unless you're cpu bound on the thread copying bytes out the first one, which seems unlikely. (Or unless your second connection is actually over a separate interface, which seems even less likely.)

        2. We drop connections when the other end dies, so clearing out the messages pending is the right thing to do. (Although here we may have FD false positives, as in CASSANDRA-3569. But if so the fix is probably along the lines Peter outlines there.)

        Show
        Jonathan Ellis added a comment - Not sure about this one: 1. Adding another TCP stream shouldn't give you more throughput or lower latency unless you're cpu bound on the thread copying bytes out the first one, which seems unlikely. (Or unless your second connection is actually over a separate interface, which seems even less likely.) 2. We drop connections when the other end dies, so clearing out the messages pending is the right thing to do. (Although here we may have FD false positives, as in CASSANDRA-3569 . But if so the fix is probably along the lines Peter outlines there.)
        Hide
        Vijay added a comment -

        I think more in parallel after retrieving from the queue are always better, specially with all kinds of delay's introduced by the networks across the DC's/Public internet can make it un-predictable, i will do some micro benchmark soon. Atleast will help the cases where there frequent disconnects and we wont need to wait till the connection is established again.

        Show
        Vijay added a comment - I think more in parallel after retrieving from the queue are always better, specially with all kinds of delay's introduced by the networks across the DC's/Public internet can make it un-predictable, i will do some micro benchmark soon. Atleast will help the cases where there frequent disconnects and we wont need to wait till the connection is established again.
        Hide
        Vijay added a comment - - edited

        Finally had a chance to do this bench mark.

        Configuration: M2.4xl (AWS)
        Traffic Between: US and EU
        Open JDK, CentOS 5.6

        3 Tests where done where the active queue is the limiting factor for the traffic to go across the nodes. Latency is the metric which we are trying to measure in this test (With 1 connection the latency is high, because of the Delay over the public internet in a AWS multi region setup).

        Code for the benchmark is attached with this ticket.

        Server A (US): java -jar Listener.jar 7103
        Server B (EU): java -jar RunTest.jar 1 107.22.50.61 7103 500

        Server C (US): java -jar Listener.jar 7103
        Server D (EU): java -jar RunTest.jar 2 107.22.50.61 7103 500

        Data is collected with 1 Second interval (plz see code for details).
        Code for the IncomingTcpConnection and OutboundTcpConnection was modified a little bit to work independent of other cassandra services (Plz see code for details).

        Show
        Vijay added a comment - - edited Finally had a chance to do this bench mark. Configuration: M2.4xl (AWS) Traffic Between: US and EU Open JDK, CentOS 5.6 3 Tests where done where the active queue is the limiting factor for the traffic to go across the nodes. Latency is the metric which we are trying to measure in this test (With 1 connection the latency is high, because of the Delay over the public internet in a AWS multi region setup). Code for the benchmark is attached with this ticket. Server A (US): java -jar Listener.jar 7103 Server B (EU): java -jar RunTest.jar 1 107.22.50.61 7103 500 Server C (US): java -jar Listener.jar 7103 Server D (EU): java -jar RunTest.jar 2 107.22.50.61 7103 500 Data is collected with 1 Second interval (plz see code for details). Code for the IncomingTcpConnection and OutboundTcpConnection was modified a little bit to work independent of other cassandra services (Plz see code for details).
        Hide
        Vijay added a comment -

        My Conclusion of the test is, Multiple connections makes sense only while talking over Multiple DC's... Within a AWS Region/DC the latencies/Delays aren't that much and is not noticeable... but it definitely is noticeable in the Multi-DC setup.

        Show
        Vijay added a comment - My Conclusion of the test is, Multiple connections makes sense only while talking over Multiple DC's... Within a AWS Region/DC the latencies/Delays aren't that much and is not noticeable... but it definitely is noticeable in the Multi-DC setup.
        Hide
        Peter Schuller added a comment -

        I just wanted to chime in and say that multiple TCP connections definitely do make a difference on high-latency connections; this is empirically obvious again and again when performing bulk transfers non-locally across the internet and needing to up the concurrency to achieve throughput.

        There are claims that TCP properly tuned will work properly with high latencies over the internet, but all my practical experience still is that the easiest way to get good throughput across arbitrary endpoints over the internet is just just up the concurrency. Whether or not TCP theoretically works under feasibly optimal circumstances is up for debate, but it seems to me that in practice multiple connections is an easy solution which pretty much always works. It should also translate well to a variety of operating system environments and not require detailed kernel tuning/knowledge to make happen.

        Show
        Peter Schuller added a comment - I just wanted to chime in and say that multiple TCP connections definitely do make a difference on high-latency connections; this is empirically obvious again and again when performing bulk transfers non-locally across the internet and needing to up the concurrency to achieve throughput. There are claims that TCP properly tuned will work properly with high latencies over the internet, but all my practical experience still is that the easiest way to get good throughput across arbitrary endpoints over the internet is just just up the concurrency. Whether or not TCP theoretically works under feasibly optimal circumstances is up for debate, but it seems to me that in practice multiple connections is an easy solution which pretty much always works. It should also translate well to a variety of operating system environments and not require detailed kernel tuning/knowledge to make happen.
        Hide
        Vijay added a comment -

        To add to the mix: co-worker pointed me to this paper... http://www-personal.umich.edu/~hacker/IPDPS.pdf

        Increasing the TCP window sizes and buffer sizes had little or no effect on the delays(/dropps) in this test.

        Show
        Vijay added a comment - To add to the mix: co-worker pointed me to this paper... http://www-personal.umich.edu/~hacker/IPDPS.pdf Increasing the TCP window sizes and buffer sizes had little or no effect on the delays(/dropps) in this test.

          People

          • Assignee:
            Unassigned
            Reporter:
            Vijay
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:

              Development